Evaluating Bitrouter Routes

---
name: evaluating-bitrouter-routes
description: Use when evaluating BitRouter route decisions or Eval Exchange subjects with task-native verifiers, human reviewers, private enterprise evaluators, agentic judges, or genuinely uncategorized evaluator sources.
---

# Evaluate BitRouter Routes

Evaluate outcomes outside BitRouter's serving path. Produce an immutable result
and stop after BitRouter reports its admission status. Do not run the optimizer
or use low-level policy publication from the evaluator workflow.

Read [the Eval Exchange reference](references/eval-exchange.md) before forming
a subject or result. It is the exact current wire and authority contract.

## Classify the evaluation

Choose scope from the observable outcome boundary:

| Evidence boundary | Scope |
|---|---|
| One request-local outcome | `request` |
| Bounded multi-request workflow or conversation | `episode` |
| Externally defined task identity plus terminal task or verifier outcome | `task` |

Choose `evaluator.kind` from the actual source:

| Evaluation source | Kind |
|---|---|
| Task-native verifier | `task_native` |
| Human reviewer | `human` |
| Private enterprise evaluator | `enterprise` |
| Agentic judge | `agentic` |
| Genuinely uncategorized evaluator | `generic` |

## Build the evaluator packet

1. Copy every decision's `decision_id`, `policy`, `route_projection`,
   `request_key`, `selected_tier`, `baseline_tier`, `policy_digest`, and
   optional `experiment` object from router-authored evidence. Preserve the
   experiment object verbatim; never invent or edit its id, arm, assignment
   unit, assignment-id digest, or challenger propensity.
2. Redact evidence before it leaves its private source. Retain raw messages,
   tool arguments, code, and evaluator output with the evaluator; place safe,
   content-addressed evidence items in the subject.
3. List only dimensions the evaluator was asked to judge. Leave unsupported
   dimensions absent. Use `inconclusive` when evidence cannot support a
   verdict.
4. Set `confidence_ppm` to the evaluator's confidence that its verdict is
   correct. Use `null` when the evaluator or rubric does not supply confidence.
   For a task or episode cost result, submit the complete unit cost as
   `cost.usd_micros` with unit `micro_usd`; never substitute one request's
   price for the complete task or episode.
5. Write a draft subject with an empty `evidence_digest`, then seal it:

   ```bash
   bro eval subject seal subject-draft.json --output subject.json
   ```

6. For a multi-decision subject, derive `decision_credit` from the fixed
   evaluator credit policy:
   - Exact supported decision/metric mappings: emit only those mappings.
   - No policy or no exact mapping: use `{}` or omit the serde-defaulted field.
     The result remains a record but produces no per-route evidence.
   For a one-decision subject, empty credit means implicit full credit. When an
   inconclusive evaluator intentionally withholds attribution, emit that
   decision with `weight_ppm: 0` instead.
   Keep hypothetical or illustrative weights outside submit-ready JSON.

## Submit and hand off

1. Insert the sealed subject and submit a result that repeats its exact
   `eval_id` and `evidence_digest`.

   ```bash
   bro eval subject put subject.json --config bitrouter.yaml
   bro eval result submit result.json --config bitrouter.yaml
   ```

2. Treat an `admitted` response as eligible evidence. Preserve `held_out`,
   `rejected`, and `disputed` responses as non-training records.
3. Hand off the sealed subject, result, submission response, and private
   evidence references, then stop. A later `bro optimize run` invocation
   is a separate autonomous authorization; do not review, publish, or run it as
   part of evaluation.

## Keep the packet consistent

- Use `subject seal` for canonical evidence hashing and JSON.
- Attribute metrics only to evidence-supported decisions.
- Never copy a task- or episode-level reward onto each request. Use a fixed
  causal policy (for example, a matched control plus one changed route family)
  or withhold credit.
- Preserve the router-authored baseline and selected tiers.
- Preserve router-authored experiment references exactly.
  Optimizer membership never comes from the evaluator-owned `cohort` string.
- Treat `inconclusive` as zero quality evidence even if an old or malformed
  packet assigns positive quality credit. Attribute cost or latency separately.
- Keep evaluator identity, rubric/config digest, evidence references,
  confidence, and idempotency key stable for an equivalent retry.
- Keep `eval_id`, result, and evidence identities attempt-specific. For task
  scope, keep `subject_id` stable for the canonical task inside its explicit
  run/source/policy namespace so repeated attempts cannot inflate the generic
  compiler's independent-task count.