Tam Eval

---
name: tam-eval
description: Evaluates LLMs on automated unit test maintenance tasks, including test creation, repair, and updating across Python, Java, and Go. Probes context-aware reasoning, code generation, and the ability to dynamically adapt tests to production code changes. Use when the user wants to benchmark on TAM-Eval, or asks about evaluating this task. Reports mutation_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.18241
  bibtex_key: bruches2026tam_eval
  confidence: high
---

# tam-eval

> TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance — Bruches et al. (2026) (arXiv:2601.18241, 2026)

## What this evaluates

Evaluates LLMs on automated unit test maintenance tasks, including test creation, repair, and updating across Python, Java, and Go. Probes context-aware reasoning, code generation, and the ability to dynamically adapt tests to production code changes.

## Datasets

- **TAM-Eval** — total 1539; splits: test (1539); repo https://github.com/trndcenter/TAM-Eval

## Metrics

- `pass_rate` — range: percent
  - Percentage of test cases that execute successfully without compilation or runtime errors.
- `line_coverage` — range: percent
  - Percentage of executable lines in the focal function covered by the generated test suite.
- `mutation_score` **(primary)** — range: percent
  - Percentage of syntactically or semantically mutated code variants (mutants) killed by the generated test suite.

## Input / output format

**Input**: Focal production file (and optionally a baseline test file for repair/update tasks), along with task-specific instructions (creation, repair, or updating).

**Output**: A complete test file (code) containing test cases for the focal function.

## Scoring recipe

```python
def evaluate(prediction_code, focal_code, task_type):
    if not compile(prediction_code): return {"pass_rate": 0, "line_coverage": 0, "mutation_score": 0}
    pass_rate = run_tests_and_check_stability(prediction_code)
    line_coverage = measure_line_coverage(prediction_code, focal_code)
    mutation_score = kill_mutants(prediction_code, focal_code)
    return {"pass_rate": pass_rate, "line_coverage": line_coverage, "mutation_score": mutation_score}
```

## Common pitfalls

- Models may generate syntactically valid tests that pass but fail to improve coverage or mutation score (superficial coverage).
- Flaky or non-deterministic tests can artificially inflate pass rates; the protocol requires consistent execution across two runs.
- High comment density in focal files can cause data leakage, leading models to generate tests based on documentation rather than executable logic.

## Evidence (verbatim from paper)

> Test coverage level. We exclude focal-test file pairs where the original test suite achieves less than 40% line coverage of the corresponding focal function. This ensures that retained test suites are not trivially under-specified and that there is a meaningful baseline from which to assess improvements in coverage and mutation score.

## Citation

```bibtex
@misc{bruches2026tam_eval,
  title={TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance},
  author={Bruches et al. (2026)},
  year={2026},
  note={arXiv:2601.18241}
}
```

- arXiv: 2601.18241