Captrack Eval

---
name: captrack-eval
description: This evaluation framework probes systematic capability drift and forgetting in large language models after post-training. It measures degradation across latent competence (knowledge, reasoning), default behavioral preferences (refusal, verbosity, formatting), and protocol compliance (instruction following, tool use, citation) in legal and medical domains. Use when the user wants to benchmark on CapTrack Evaluation Suite, or asks about evaluating this task. Reports average forgetting.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.06610
  bibtex_key: thede2026captrack
  confidence: high
---

# captrack-eval

> CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training — Thede et al. (2026) (arXiv:2603.06610, 2026)

## What this evaluates

This evaluation framework probes systematic capability drift and forgetting in large language models after post-training. It measures degradation across latent competence (knowledge, reasoning), default behavioral preferences (refusal, verbosity, formatting), and protocol compliance (instruction following, tool use, citation) in legal and medical domains.

## Datasets

- **CapTrack Evaluation Suite** — total ?; splits: test (-1); repo https://github.com/thomsonreuters/captrack

## Metrics

- `average forgetting` **(primary)** — range: percent
  - Percentage drop in performance from the pre-training baseline to the post-trained model, calculated per capability category. Computed as (baseline_score - post_training_score) / baseline_score * 100.

## Input / output format

**Input**: Task-specific prompts or questions from the CapTrack suite, covering legal and medical domain tasks across competence, behavioral, and protocol capability groups.

**Output**: Model-generated text response.

## Scoring recipe

```python
def compute_forgetting(pre_scores, post_scores):
    forgetting_rates = []
    for pre, post in zip(pre_scores, post_scores):
        if pre > 0:
            forgetting_rates.append((pre - post) / pre * 100)
        else:
            forgetting_rates.append(0.0)
    return sum(forgetting_rates) / len(forgetting_rates)
```

## Common pitfalls

- Forgetting is defined relative to a pre-training baseline, not absolute post-training performance; reporting only post-training scores misses the drift.
- The suite aggregates results across heterogeneous capability groups (CAN, WILL, HOW) that use different evaluation metrics (e.g., accuracy vs. refusal rates vs. formatting checks), so unstratified averaging obscures specific failure modes.

## Evidence (verbatim from paper)

> We repeat each evaluation with three random seeds and report the average results. We report average forgetting within each capability category, with faint markers indicating benchmark-level variability.

## Citation

```bibtex
@misc{thede2026captrack,
  title={CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training},
  author={Thede et al. (2026)},
  year={2026},
  note={arXiv:2603.06610}
}
```

- arXiv: 2603.06610