Flan Heldout Instruction Evaluator

---
name: flan_heldout_instruction_evaluator
description: Evaluate held-out direct and CoT instruction examples, compute accuracy delta, and emit FLAN recovery mechanism checks.
---

# FLAN Held-Out Instruction Evaluator

Use this skill after an instruction-finetuning run to measure held-out generalization.

## Inputs

- Held-out examples and predictions before/after finetuning.
- Mixture audit, training trace, and target metadata.

## Outputs

- Accuracy before/after/delta and mechanism checks.

## Workflow

1. Check held-out ids are absent from training ids.
2. Compute exact-match accuracy before and after.
3. Emit mechanism checks for CoT, optimizer, and target matching.

## Validation

Run `python tests/test_evaluator.py`.

## Limitations

Supports deterministic proxy metrics, not full benchmark evaluation.