Agent Development

---
name: agent-development
description: Use when building, enhancing, or testing the NPA chat agent backend — grounded-first routing, cost-aware Token Factory model selection, the embedded-backend mechanism, and cheap-token test tiers.
---

# Agent Development (cheap-token chat backend)

How to *develop* the NPA agent chat backend. For *operating* a deployed agent
(deploy/bootstrap/verify, chat UX, API grounding, Rerun) use
`skills/tools/npa-agent/SKILL.md`; for fresh deploy/teardown loops use
`skills/workflows/agent-fresh-operate/SKILL.md`.

Guiding principle: **the cheapest token is the one you never spend.** Protect and
widen the zero-token grounded path; make the unavoidable model calls small,
structured, and cheap-model-first.

## Architecture (three layers)

1. **Grounded intent router (zero tokens)** — `npa/src/npa/cli/agent_chat.py`.
   `match_chat_intent()` classifies a turn; `build_grounded_reply()` answers from
   live session state with **no model call** (`"grounded": true`). Most operator
   turns end here.
2. **Cost-tier routing (cheap model calls)** — `npa/src/npa/cli/agent_routing.py`.
   Only turns that fall through the router reach Token Factory, and this layer
   keeps them cheap.
3. **Token Factory client** — `npa/src/npa/clients/token_factory.py` (operator/SDK
   path) and the embedded `_provider_chat` / `_chat_with_resilience` in
   `npa/src/npa/cli/agent.py` (agent-VM path).

## The embedded-backend mechanism (critical)

The agent VM runs `backend.py`, which is built as one big f-string inside
`_bootstrap_agent_stack` in `npa/src/npa/cli/agent.py`. Pure helper modules are
inlined into it via placeholder substitution — the pattern to reuse when adding
logic:

- Real module (normal Python, **no** brace escaping): `agent_chat.py`,
  `agent_workflow.py`, `agent_routing.py`.
- Embed helper `_embedded_agent_<name>_source()` strips the docstring +
  `from __future__` line.
- Placeholder constant `_AGENT_<NAME>_EMBED` appears in the template and is
  replaced in the `.replace(...)` chain near the end of `_bootstrap_agent_stack`.

Rules:

- Put testable logic in a **real module** and embed it; do **not** write new
  logic directly inside the f-string unless it must touch template variables.
- Code written *inside* the f-string must escape literal braces as `{{` / `}}`
  and newlines in strings as `\\n`; substitutions use single `{var}`.
- After editing the template, **validate the rendered backend compiles** (see
  Testing). A stray brace is a `SyntaxError` at import of `agent.py`.
- **Embedded routes register before the template's own routes.** The
  `_AGENT_*_EMBED` placeholders sit earlier in the f-string than most `@app.*`
  blocks, and Starlette resolves the **first** matching route, so an embedded
  module silently wins over a same-path handler written further down in
  `agent.py`. `/artifacts/file/{filename}` and `/artifacts/download` each carried
  two definitions this way, and the shadowed copies in `agent.py` were the weaker
  ones — no `Content-Disposition`/`nosniff` headers, no run-scoped inventory
  authorization — so the file a maintainer would open described a contract the
  deployment did not serve. Before adding a route, grep the embedded modules for
  its path; `test_rendered_backend_registers_no_shadowed_routes` fails the build
  on any method+path registered twice.

## Cost-tier routing (`agent_routing.py`)

Pure, side-effect-free functions (no network) so they unit-test cheaply:

- `classify_tier(text, intent, messages)` → `cheap` (default) / `standard`
  (long/compound) / `reasoning` (analytical language) / `vision` (image content).
- `build_model_ladder(tier, configured, interactive, requested_model,
  allow_tier_defaults)` → cheapest-capable first; explicit user model wins;
  operator allowlist (`NPA_AGENT_LLM_MODELS`) respected when set.
- `flavor_variants` + `filter_available` — use only documented concrete model
  IDs and drop unavailable defaults. The current public replacements have no
  verified `-fast` variants; never invent one. Explicit selections are tried
  even when absent from the model list (dedicated endpoints can differ).
- `chat_extra` / `thinking_enabled` — disable hidden reasoning traces off the
  reasoning tier (don't pay for discarded tokens).
- `enforce_input_budget` — cap oversized pastes (head+tail preserved).
- `usage_summary` — surface per-turn token usage.

The `/chat` handler classifies the tier, enforces the input guardrail, honors an
explicit model override, and returns `tier` + `usage` + `input_budget_ok`.

## Adding a capability cheaply (decision order)

1. **Grounded intent** — can a regex intent + grounded state reply answer it?
   (0 tokens) Add to `_INTENT_RULES` / `build_grounded_reply` in `agent_chat.py`
   and cover it in `test_agent_chat.py`. Prefer this for anything high-frequency.
   Three invariants apply to every new or edited rule:
   - **Add an `INTENT_APIS` entry.** It is not only reply metadata:
     `_semantic_route` builds the semantic fallthrough's `known_intents` from its
     keys, so an intent missing from it is unreachable by any paraphrase the
     regex misses and its grounded replies report an empty `apis_used`.
     `test_every_intent_declares_its_apis` fails the build otherwise.
   - **Earlier rules win**, and `match_chat_intent` applies several hard-coded
     checks before the list at all. Adding a phrasing an earlier rule already
     claims silently changes nothing; check what currently matches first.
   - **Keep sibling rules symmetric.** The tool-capability rules are near
     copies, so a verb added to one belongs in all of them. "what can lancedb
     do" degraded to the generic component reply for exactly this reason while
     the identical Sonic/LeRobot/Genesis wording stayed tool-specific.
   A phrasing that matches nothing is not neutral — it falls through to a paid
   model call for an answer the grounded layer already had.
2. **Cheap model** — if it needs generation, let routing pick the cheap tier;
   only add reasoning/vision signals to `classify_tier` when the turn truly
   needs them.
3. **Escalate deliberately** — reserve `MiniMaxAI/MiniMax-M3` for
   analytical/physical-AI/vision turns; it is overkill for routine chat.
4. **Visual feedback** — UI **Describe this** captures the active viewer frame
   and posts multimodal `/api/chat` with `visual_context`. Helpers live in
   `agent_visual_feedback.py` (embedded). Never ground these turns; use vision
   when an image is attached. See `skills/atomic/agent-visual-feedback/SKILL.md`.

## Token Factory notes

- OpenAI-compatible: base `https://api.tokenfactory.nebius.com/v1/`, key
  `NEBIUS_TOKEN_FACTORY_KEY`. Same `chat/completions` shape everywhere.
- Public defaults follow the [August 2026 migration notice](https://docs.tokenfactory.nebius.com/august-2026-deprecation-notice):
  `nvidia/Nemotron-3_5-Lightning` for cheap/standard text,
  `MiniMaxAI/MiniMax-M3` for reasoning/vision. No default `-fast` IDs are added.
  Vision never falls back to the text-only Lightning default.
- Cost-ordered default ladder lives in `DEFAULT_LLM_MODELS` (cheap first). A bare
  `npa agent deploy` seeds this whole ladder, so per-turn routing reaches every
  tier without `--llm-models`. Explicit `--llm-models` is a governance allowlist;
  `/api/models` still surfaces every model the key can serve for per-request
  selection.
- Reasoning-trace handling: `split_reasoning()` normalizes Cosmos3 inline
  `<think>` and Kimi/GLM `reasoning` fields.
- `chat_extra(tier, model)` selects thinking parameters per attempted model:
  Lightning uses `chat_template_kwargs.enable_thinking=false`; MiniMax uses
  `chat_template_kwargs.thinking_mode="disabled"`. Analytical turns retain
  reasoning, and unknown custom models receive no guessed template options.

`/api/models` and `/api/session` share observed model availability. Prefer the
configured default only when the provider lists it as chat-capable and it is
within the explicit operator allowlist; otherwise choose the first eligible
configured fallback. Keep the actual full provider catalog for explicit model
selection. Never append an unavailable configured model to make a default look
valid. The documented `GET /v1/models?verbose=true` supplies architecture
modality; basic listings identify only known chat families, not arbitrary
embedding or custom models. Empty successful discovery reports unavailable;
failed discovery or unknown chat suitability reports unknown, with a null
default and no provider diagnostics. Grounded/session operations still work.
`test_agent_model_availability.py` exercises the rendered routes, refresh and
failure cache, allowlist, explicit selection and non-chat exclusion. A catalog
entry proves listing availability, not successful inference; retain live chat
verification after deployment.

## Testing tiers (keep tokens out of CI)

Follow `skills/atomic/testing-conventions/SKILL.md`; use `npa/.venv/bin/python`.

- **Tier 0 — pure logic (0 tokens):** `npa/tests/cli/test_agent_routing.py`
  (tiers, ladder, flavor, availability filter, budget, usage) and
  `test_agent_chat.py` (intent router, grounded replies). Highest-value coverage.
- **Tier 1 — mocked LLM (0 tokens):** patch `_provider_chat` /
  `_chat_with_resilience`; assert prompt assembly, resilience fallthrough, and
  tier/usage in the response.
- **Rendered-backend check:** confirm the embedded backend compiles with all
  wiring inlined — render `setup_script` with mocked SSH, extract the
  `backend.py` heredoc body, and `ast.parse` + `compile` it. Guards the f-string.
- **Whole-surface audit (0 tokens):**
  `npa/scripts/audit_agent_capabilities.py` goes one step further than the
  compile check — it *runs* the rendered backend against a sandbox state root and
  probes every parameterless `GET` plus every advertised intent. Use it to prove
  a change did not silently unregister a route or re-route an intent, and to
  answer "does the agent really support X" without a VM. Compiling is not
  running: an import-time failure in a shipped `agent_backend` module passes the
  compile check and fails here. `--serve-live` runs the same probes against a
  real `uvicorn` process on loopback with the deployed systemd unit's arguments,
  which is the only tier that covers lifespan and websocket-flag behavior.
- **Tier 2 — live e2e (bounded tokens):** gate behind `NPA_AGENT_CHAT_LIVE=1` /
  `NPA_INTEGRATION_E2E=1`; pin the cheapest model; assert `grounded: true` where
  possible so most turns cost 0 tokens.
- **Live public-model migration coverage:**
  `npa/tests/e2e/test_agent_token_factory_e2e.py` runs the rendered backend under
  uvicorn on loopback, sends synthetic text/reasoning/vision requests through
  `/chat`, and checks real Token Factory answers and model selection. Only the
  deployment SSH transport is replaced while rendering; inference is real.
  It uses `token_factory_e2e` and skips without configured provider credentials.

For action-loop validation, inspect the final answer against the original goal
as well as the successful tool observations. `stopped_reason="done"` can come
from a deterministic empty-result reply; it does not prove the model produced a
final answer or that every requested part was answered. An empty lookup is
terminal only when its subject and scope satisfy the whole request. A filtered
query with zero matches cannot establish an empty store, and an unrelated empty
run lookup cannot replace an available status or tool-catalog answer. Preserve
these distinctions in evaluation receipts and regression tests.

```bash
npa/.venv/bin/python -m pytest npa/tests/cli/test_agent_routing.py \
  npa/tests/cli/test_agent.py npa/tests/cli/test_agent_chat.py \
  npa/tests/smoke/test_agent_smoke.py npa/tests/smoke/test_agent_chat_smoke.py \
  npa/tests/guardrails/test_agent_secret_guard.py -q
```

## Guardrails

- Never leak credentials/auth env/secrets into chat, logs, or workflow YAML.
- Do not hardcode project IDs, tenant IDs, bucket names, registry IDs, usernames,
  or public IPs in code or examples.
- Preserve the chat contract: grounded-first, then a cheap LLM fallback. Do not
  regress the agent into a chat-only (always-LLM) design.

## Agentic surface (fallthrough beyond grounded)

Everything below runs **only after** the grounded intent router misses; the
zero-token path stays the default. Design doc:
`docs/architecture/agent-competitive-plan.md`.

- **Bounded tool-calling loop** — `npa/src/npa/cli/agent_actions.py`
  (`run_action_loop`): classify → plan → call → observe → decide → stop with a
  hard `max_steps` guard, an explicit `TOOL_ALLOWLIST`, and a confirmation-gate
  contract. GPU/destructive tools need a token **bound to the action digest**
  (`action_digest`); tokens are single-use. Route: `POST /api/agent/act`.
- **Autonomous Sim2Real drive** — `npa/src/npa/cli/agent_sim2real_loop.py`
  (`drive_sim2real_loop`): launch→status→gate→diagnose→adjust→re-run, mirroring
  the engine `promote_checkpoint`/`loop_back` gate. Every launch is
  confirmation-gated; stages complete only when live status confirms the run
  (no fabrication); stops on insufficient signal / no-adjustment to avoid
  runaway GPU. Route: `POST /api/agent/sim2real/drive`; chat intent
  `drive_sim2real` returns grounded guidance.
- **Semantic fallthrough** — `npa/src/npa/cli/agent_semantic_router.py`
  (`classify_intent_semantic`): keyword + cache (0 tokens) then one cheap
  structured call to map regex-missed paraphrases to a known intent/action.
  Wired into the `/chat` fallthrough; degrades to `none` on failure. Parity
  intents still match in `match_chat_intent` and never reach it.
- **Quantitative viewer eval + memory** — `agent_visual_feedback.py`
  (`extract_quantitative_signals`, `compare_rollouts`) and the shipped package
  `npa/src/npa/agent_backend/memory.py` (`RunMemory`, storage-injected, no
  hardcoded bucket). Routes: `GET/POST /api/agent/memory/*`.
- **Task-eval harness** — `npa/tests/agent_eval/` (mocked, 0 tokens): scenarios
  + scorecard (`success_rate`/`avg_steps`/`avg_tokens`); live variant gated on
  `NPA_AGENT_CHAT_LIVE=1`.

**Shipped vs embedded (Phase G):** new logic still uses the embed mechanism by
default; `agent_backend/` is the shipped-package migration target (uploaded to
`/opt/npa-agent/agent_backend/`, imported via `sys.path`). `agent_memory` is the
migrated pilot; `cli/agent_memory.py` is a re-export shim. Rendered-backend
compile check: `npa/tests/cli/test_agent_backend_render.py`.

## Blueprint incorporation (retrieval, observability, adversarial eval)

Open-source-only parity with the Nebius "Blueprints" reference agent — Token
Factory (LLM + embeddings) + AI Cloud only, no LangSmith/Pinecone/Tavily/
Snowglobe. Design doc: `docs/architecture/blueprint-incorporation-plan.md`.

- **Retrieval / grounding (Phase H)** — shipped `npa/src/npa/agent_backend/retrieval.py`
  (shim `cli/agent_retrieval.py`). `index_corpus()` chunks + embeds repo
  `docs/`+`skills/` (and optional live web) into an injected vector store
  (`InMemoryVectorStore` / `JsonVectorStore` / `build_lance_store` for **LanceDB**);
  `retrieve()` returns typed `Citation`s; `format_grounded_answer()` is extractive
  (0 generation tokens). `embed` and `web_search` (SearXNG-shaped, provider-
  agnostic) are injected. Read-only `retrieval_search` tool in `TOOL_ALLOWLIST`;
  routes `POST /api/agent/retrieval/index`, `GET /api/agent/retrieval/search`,
  `GET /api/agent/retrieval/status`; grounded-first `/chat` fallthrough that only
  fires when a corpus is indexed and the top match clears the score floor.
- **Observability (Phase I)** — shipped `npa/src/npa/agent_backend/trace.py`
  (shim `cli/agent_trace.py`). `spans_from_action_loop` / `spans_from_drive` wrap
  the existing step/iteration traces in structured spans emitted through an
  **injected tracer** (`NullTracer` default; `build_langfuse_tracer` /
  `build_otel_tracer` guarded). `redact_attributes` keeps secrets/PII out of
  spans. `analyze_traces` clusters traces + flags silent failures (truncation,
  empty tool results, unsurfaced errors, max-steps). Routes:
  `GET /api/agent/trace/spans`, `POST /api/agent/trace/analyze`.
- **Adversarial eval (Phase J)** — `npa/tests/agent_eval/adversarial.py`: persona
  (Token-Factory-generated, mocked in CI) × prompt-injection scenarios run
  against the real modules; `validate_output` always runs the repository-owned
  pure-Python checks. `test_agent_adversarial_scorecard.py`
  gates `defense_rate` **delta-vs-baseline** (`adversarial_baseline.json`).

**Optional extras (injected/guarded, absent degrades gracefully):**
`npa[agent-eval]` is a compatibility alias with no extra dependencies.
`npa[agent-trace]` (langfuse /
opentelemetry-sdk). The LanceDB store is now part of the base install.
Embeddings default to
`NPA_AGENT_EMBED_MODEL` (confirm with `npa workbench token-factory models`);
LanceDB URI, SearXNG URL, and tracer keys are env/config-resolved, never
hardcoded. New agentic tests: `test_agent_retrieval.py`, `test_agent_trace.py`,
and `agent_eval/test_agent_adversarial_scorecard.py`.

## Evidence-backed improvements

The shipped `agent_backend/improvements.py` queue detects explicit tool failures,
nonterminal empty results, truncation and exhausted action loops. Direct action,
Sim2Real drive and semantic-action chat record observations linked to the active
trajectory episode. Successful recovery and empty discovery that answers the
whole request are not implementation defects. Premature empty-result termination
requires separate goal-answer review even when the loop reports `done`.
The queue produces triage work packages; it does not run
shell commands, modify source, launch workers or publish code.

Enable it explicitly with `NPA_AGENT_IMPROVEMENT_CONFIG`, pointing to an
owner-only JSON file outside the checkout. Configure `directory`, `repository`,
`evidence_directory`, `reviewers`, optional `private_literals`, and `scopes`.
Each scope requires a stable `scope_id`, known `component`, exact relative
`files`, full `base_revision`, `required_checks`, supported `lesson_keys`, and
optional `version`. Queue and evidence directories must be outside the source
repository, mode 0700; configuration, database and reports are mode 0600.
Dataset URI/prefix and the configured trajectory literal denylist also apply
before queue persistence. Never put a live configuration or raw report in Git.

Use `ImprovementStore.claim` to obtain an exact work package and opaque
`claim_token`. Its field name ensures trajectory redaction removes the credential.
SQLite transactions prevent overlapping file ownership across processes; every
update requires the current generation and lease. Release only after joining
the worker. Scope comes from coordinator configuration, never an exception or
model-supplied path. Changed scope configuration requires explicit reconciliation.

The coordinator captures `begin_candidate` **before** running the scope's real
checks with `npa/.venv/bin/python`. Supply the complete candidate changed-file
list, excluding separately owned concurrent work. Feed the actual
`subprocess.CompletedProcess` and nonempty report bytes to
`write_validation_receipt`, then `record_validation`. The adapter binds the
report digest, check identity, exit code, source snapshot, scope and claim
generation. It never executes commands supplied by a finding. Failed, partial,
missing or changed evidence cannot advance to independent review.

An independently obtained review report enters through the **local trusted**
`write_review_receipt` adapter and `review`. Reviewer identities are explicitly
coordinator-attested external evidence; the agent's shared HTTP login does not
establish multiple independent reviewers. HTTP routes under
`/agent/improvements` accept existing protected receipt references, never a
`passed` flag or an asserted reviewer name as proof. Workers cannot create
receipts through HTTP. Retain the separate actor's actual report privately.

Verified lessons are fixed behavior keys with evidence references, not arbitrary
prose inserted into prompts. Current keys are
`trajectory_observation_conservation` and `inspect_failed_tool_evidence`.
Relevant skill/action context retrieves them without a model call. Reuse checks
the source snapshot and retained evidence, records consumption and action
outcome, and deactivates after a distinct recurring failure. A storage failure
returns pending feedback while preserving the completed product action; never
repeat a successful GPU operation to repair feedback. Missing opt-in config
reports disabled.

Regression checks include `test_agent_improvements.py`,
`test_improvement_episode_context.py`, and `test_agent_improvement_render.py`.
The rendered tests exercise all three action hooks and actual shipped modules.
They do not replace a deployed zero-token lifecycle exercise with independent
review evidence. The opt-in `test_agent_improvements_live.py` uses
`NPA_AGENT_IMPROVEMENT_LIVE=1`, a dedicated approved component supplied through
`NPA_AGENT_IMPROVEMENT_LIVE_COMPONENT`, and an owner-only lifecycle bundle at
`NPA_AGENT_IMPROVEMENT_LIVE_BUNDLE`. The coordinator prepares that bundle from
actual check receipts and a separately obtained review; the live test cannot
invent either. Use the existing private live-agent credential configuration.

## Source Layout

- CLI + bootstrap + embedded backend: `npa/src/npa/cli/agent.py`
- Grounded intent router (testable): `npa/src/npa/cli/agent_chat.py`
- Cost-tier routing (testable): `npa/src/npa/cli/agent_routing.py`
- Agentic tool loop / sim2real drive / semantic router:
  `npa/src/npa/cli/agent_actions.py`, `agent_sim2real_loop.py`,
  `agent_semantic_router.py`
- Shipped backend package: `npa/src/npa/agent_backend/` (memory pilot)
- Token Factory client: `npa/src/npa/clients/token_factory.py`
- Routing tests: `npa/tests/cli/test_agent_routing.py`; agentic tests:
  `test_agent_actions.py`, `test_agent_sim2real_loop.py`,
  `test_agent_semantic_router.py`, `test_agent_memory.py`,
  `test_agent_backend_render.py`; eval harness: `npa/tests/agent_eval/`