Codetracer Eval

---
name: codetracer-eval
description: This benchmark evaluates an agent's ability to localize the onset of failure within long-horizon code execution trajectories by analyzing heterogeneous run artifacts. It probes how well models can distinguish genuinely failure-relevant steps from salient but irrelevant logs, diagnose execution bottlenecks, and recover from early wrong commitments under constrained token budgets. Use when the user wants to benchmark on CodeTraceBench, or asks about evaluating this task. Reports step-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11641
  bibtex_key: [REDACTED]
  confidence: high
---

# codetracer-eval

> CodeTracer: Towards Traceable Agent States — Han Li et al. (2026) (arXiv:2604.11641, 2026)

## What this evaluates

This benchmark evaluates an agent's ability to localize the onset of failure within long-horizon code execution trajectories by analyzing heterogeneous run artifacts. It probes how well models can distinguish genuinely failure-relevant steps from salient but irrelevant logs, diagnose execution bottlenecks, and recover from early wrong commitments under constrained token budgets.

## Datasets

- **CodeTraceBench** — total 3326; splits: test (3326); repo https://github.com/NJU-LINK/CodeTracer

## Metrics

- `step-level Precision` — range: percent
  - TP / (TP + FP), where TP and FP are correctly and incorrectly identified failure steps at the step level.
- `step-level Recall` — range: percent
  - TP / (TP + FN), where FN are missed failure steps.
- `step-level F1` **(primary)** — range: percent
  - 2 * (Precision * Recall) / (Precision + Recall), macro-averaged across difficulty, category, and execution stage splits.
- `token usage` — range: other
  - Total tokens consumed by the model during the localization and replay phases.

## Input / output format

**Input**: Heterogeneous run directories/logs containing execution artifacts from agent frameworks, optionally pre-processed into standardized traces or hierarchical trace trees.

**Output**: A set of identified step indices or stage labels corresponding to the earliest critical failure point(s) in the trajectory.

## Scoring recipe

```python
def compute_step_metrics(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
# Macro-average across splits (Easy/Hard, categories, stages)
```

## Common pitfalls

- Confusing salient or noisy log artifacts with genuinely failure-relevant steps, leading to high recall but low precision.
- Token budget constraints directly dictate search depth; models may terminate prematurely on hard tasks, missing dispersed failure evidence.
- Step-level granularity requires precise alignment between predicted steps and ground-truth annotations, which can be sensitive to trajectory parsing variations.

## Evidence (verbatim from paper)

> We report macro averaged step level Precision, Recall, and F1 (formal definitions in [Appendix C]) and total token usage. Across all backbones, adding structure and tracing signals substantially improves localization quality over raw log prompting. The three frontier backbones reach comparable F1 (46–48%) but diverge in how they traverse the trace.

## Citation

```bibtex
@misc{hanli2026codetracer,
  title={CodeTracer: Towards Traceable Agent States},
  author={Han Li et al. (2026)},
  year={2026},
  note={arXiv:2604.11641}
}
```

- arXiv: 2604.11641