Omh Llm App Dev

---
name: "omh-llm-app-dev"
description: "[omh] Hermes LLM App Development workflow: prepare a build handoff for an LLM-powered feature with a pinned provider boundary, schema-first outputs, versioned prompt files, grounded retrieval, and an eval suite as a shipped deliverable. Use when the user says: llm-app-dev, llm app development, llm application development, build an llm app, build an llm feature, llm feature development, build a rag pipeline, rag pipeline."
metadata:
  hermes:
    tags: [workflow, oh-my-hermes, delivery]
    category: delivery
    phase: llm-app-dev
    role: operator
    quality_tier: delivery-gated
---

# Llm App Dev

This is a Hermes-native `llm-app-dev` workflow skill.

## Why This Exists

`llm-app-dev` exists because the failure modes of an LLM feature are not the failure modes of the code around it. A floating model alias, a prompt buried in a string literal, an output scraped out of prose with a regex, and a retrieval layer nobody measured all pass code review and all fail in production, and without a golden set nobody can tell whether the next prompt edit helped or hurt.

## Do Not Use When

- The subject is comparing executors or agent harnesses - Codex against Claude Code against Hermes coding - rather than evaluating the product's own model calls; use `agent-evaluation`.
- An agent run is already stuck, looping, or drifting and needs diagnosis; use `agent-debug`.
- The subject is the harness's own context window, prompt caching, or token budget rather than the application being built; use `context-budget-review`.
- The request is a prompt-injection, secret-handling, or dependency risk gate on work that already exists; use `security-safety-review`.
- The feature makes no model call - the LLM is only mentioned as the subject being discussed - so this is a direct answer, not a build handoff.

## Examples

Good example:

- Prompt: $llm-app-dev we are adding an invoice-field extractor that calls a model per upload - set it up so we can change the prompt later without guessing.
- Expected behavior: Name the rails, put the provider call behind one client module with a pinned model ID, declare the extraction schema and the repair path, lay the prompt out as a versioned file, and specify the golden set and validators that let the next prompt edit be compared against this baseline.
- Why: The feature is a real model call whose output another system consumes, which is exactly where an unpinned model, an inline prompt, and a missing golden set become expensive later.

Bad example:

- Prompt: $llm-app-dev the extractor is done - confirm the new prompt is better than the old one.
- Expected behavior: Prepare the paired baseline-vs-candidate comparison and state that no result exists until the run is observed; report nothing about which prompt is better.
- Why: Better is a claim about an observed run. Without one, the comparison is a design, and calling it a result is the false-green this workflow exists to prevent.

## Completion Checklist

- Every rail - provider boundary, structured output, prompt artifacts, retrieval grounding, evaluation - is either decided or explicitly deferred with a reason.
- One client boundary owns the model ID, credentials, timeout, retry, and backoff, and no credential appears in source, prompts, tests, or examples.
- The model ID is exact, and it is recorded next to any result meant to be compared.
- Every model response is validated against a declared schema, with a bounded repair path and a loud failure - no prose scraping.
- Prompts are files with a version identifier, and system rules, task instruction, and injected context are separated.
- Untrusted retrieved or user-supplied content is fenced from the instruction channel and cannot change the task.
- The eval deliverables - golden set, task-level validators, baseline-vs-candidate comparison - exist as committed artifacts, and retrieval is evaluated before generation when retrieval is in the path.
- Token, latency, and cost figures come from an observed run or stay null; no design output is reported as an eval result, implementation, review, CI, or merge evidence.

## Recovery Notes

- If the exact model ID or provider is not decided yet, name the candidates and prepare the boundary against a config value rather than choosing one silently.
- If no failing case can be stated, the golden set has no seed: collect the real failures first, because a golden set written from imagination measures the imagination.
- If a response cannot be made to satisfy the schema after one bounded repair, treat that as a schema or prompt defect and record it as a golden-set case rather than loosening validation.
- If retrieval quality was never measured, stop before scoring generation and route the retrieval evaluation first; a generation score on unmeasured retrieval is not attributable.
- If the comparison run did not emit tokens or cost, leave those fields null and say the harness did not report them; never reconstruct them from pricing tables.

## Workflow Lane

- Current lane: **Coding handoff** (`idea-to-deploy`, `llm-app-dev`, `cto-loop`, `deploy-and-monitor`, `code-review`, `build-failure-triage`, `verification-gate`, `security-safety-review`, `+13 more`) - coding owners, handoffs, review, CI, and merge evidence.
- If intent belongs to another lane, hand back to `oh-my-hermes` or name the adjacent workflow.
- Shared product, routing, compatibility, and evidence rules: `omh-routing/references/skill-common-rail.md`.

## Use When

Use when the work is building or hardening an LLM-powered feature - provider calls, structured outputs, prompt files, retrieval grounding, or the eval suite that guards a prompt or model swap - and the request needs engineering discipline before a coding handoff.

    Strong routing signals: `llm-app-dev`, `$llm-app-dev`, `llm app development`, `llm application development`, `build an llm app`, `build an llm feature`, `llm feature development`, `build a rag pipeline`, `rag pipeline`, `retrieval augmented generation`, `structured output schema`, `json schema output`, `prompt versioning`, `llm eval suite`, `golden set`, `LLMアプリ開発`, `LLM機能開発`, `RAGパイプライン構築`, `構造化出力スキーマ`, `プロンプトのバージョン管理`, `LLM評価セット`, `llm 앱 개발`, `llm 애플리케이션 개발`, `llm 기능 개발`, `rag 파이프라인`, `rag 파이프라인 구축`, `구조화된 출력 스키마`, `프롬프트 버전 관리`, `llm 평가셋`, `골든셋`, `大模型应用开发`, `检索增强生成`, `结构化输出模式`, `提示词版本管理`, `评测集`

## Catalog Metadata

Category: `delivery`
Phase: `llm-app-dev`
Hermes role: `operator`
Quality tier: `delivery-gated`
Reasoning demand: `heavy`

Quality bar:

- Decide the rails in order - provider boundary, structured output, prompt artifacts, retrieval grounding, evaluation - and say which are deferred rather than leaving them unnamed. Load `references/build-rails.md` for the per-rail decision and its failure mode.
- Route every provider call through one client boundary module that owns the model ID, credentials, timeout, retry policy, and rate-limit backoff. A second call site that builds its own client is how a model pin, a timeout, and a retry policy quietly diverge.
- Pin the exact model ID as a named constant or config value, never a floating alias, and record it next to any result that will be compared to another result.
- Take structured output from a declared schema - a JSON schema, a typed parser, or the provider's structured-output mode - and validate every response against it. A response that fails validation is repaired by one bounded re-ask that shows the validation error, then fails loudly; it is never regex-scraped out of prose.
- Keep prompts as reviewable files with a version identifier, separated into system rules, task instruction, and injected context, so a prompt change shows up in a diff instead of inside a string literal.
- For retrieval, fix chunking and citation grounding first and evaluate retrieval before evaluating generation: a generation score on top of unmeasured retrieval cannot tell a bad answer from a bad document set.
- Ship the eval suite as a deliverable, not a follow-up: golden set, task-level validators, baseline-vs-candidate comparison, with deterministic validators wherever the task allows one. Load `references/eval-harness.md` for the golden-set shape, the validator ladder, and the comparison record.
- Run the regression before a prompt or model swap, not after, and compare baseline against candidate on the same golden set with token and cost capture. Report only what the run reported; a metric the harness did not emit stays null.
- Give every agentic loop its budgets as product features, not prompt advice: step, time, token, cost, and tool-call budgets each with a recorded termination reason, and for recursive delegation the budgets bind the whole tree, not each node separately.
- Separate draft from commit for risky side effects: reads and drafts may run autonomously when scoped and labeled, but external writes, deletions, and communications need an approval record outside the prompt - a model's stated intention is never the authorization.
- Keep design and evidence separate: a prepared schema, prompt layout, or eval plan is not implementation, an observed eval run, review, CI, or merge evidence.

Handoff policy:

Keep the rail choices, schema shape, prompt-artifact layout, and eval design in Hermes as a prepared build handoff. Prepare a selected executor/runtime handoff for the code itself, and record provider calls, eval runs, token counts, and cost only from observed run artifacts.

Required inputs:

- the feature the model is supposed to perform
- the exact provider and model ID under consideration
- the shape of the output the caller consumes
- the failing cases that must not regress

Expected outputs:

- rail decisions across provider boundary, structured output, prompt artifacts, retrieval grounding, evaluation
- the output schema and the validate-and-repair path for a response that does not match it
- the prompt artifact layout and its version identifier
- the eval deliverables - golden set, task-level validators, baseline-vs-candidate comparison
- the executor handoff and what stays unobserved until a run produces it

Artifact expectations:

- prompt files committed under version control with a version identifier the call site records, so a response can be traced to the prompt that produced it
- a golden set committed beside the code as data, not as prose in a chat log

Safety rules:

- Do not hardcode an API key, token, or provider credential in source, prompts, tests, or examples; the client boundary reads them from the environment or a secret store.
- Do not pin a model by a floating alias when the behavior is being evaluated; a benchmark against a moving target proves nothing.
- Do not catch provider failures broadly; classify timeout, rate limit, transient server error, invalid request, and content refusal separately, because only some of them are safe to retry.
- Do not put untrusted content - retrieved documents, user uploads, tool output, web pages - in the same channel as instructions, and never let it change the task.
- Do not report token counts, latency, or cost that a run did not produce; telemetry the run did not report stays null and is never estimated.
- Do not claim an eval passed, a prompt shipped, or a model swap is safe from a prepared design; every such claim needs an observed run.

## Runtime Evidence

Preferred harness for this skill: `coding-handling`.

```sh
omh runtime record --skill llm-app-dev --harness coding-handling --status started
```

Record observed delegation results; otherwise return `not_available` or `not_observed`.
Prepared OMH routing is not execution, review, CI, merge-readiness, or merge evidence.
- Treat wrapper memory/context summaries as advisory local context, not proof of opaque Hermes memory reads or changes.
Preserve workflow intent and stop conditions; verify before claiming completion.

Use Hermes-native subagent/delegation features when available: native subagents -> Hermes delegation when available, otherwise sequential lanes.

Shared product, compatibility, topology, memory, harness, and execution rules: `omh-routing/references/skill-common-rail.md`. Load it when applicable; otherwise name an unavailable capability.