Taco Eval

---
name: taco-eval
description: Evaluates code generation models on competition-level algorithmic programming problems. It probes fine-grained capabilities across different programming skills and difficulty levels by measuring whether generated Python programs correctly solve given problems under strict constraints. Use when the user wants to benchmark on TACO, or asks about evaluating this task. Reports pass@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.14852
  bibtex_key: li2023taco
  confidence: high
---

# taco-eval

> TACO: Topics in Algorithmic COde generation dataset — Li et al. (2023) (arXiv:2312.14852, 2023)

## What this evaluates

Evaluates code generation models on competition-level algorithmic programming problems. It probes fine-grained capabilities across different programming skills and difficulty levels by measuring whether generated Python programs correctly solve given problems under strict constraints.

## Datasets

- **TACO** — total 26443; splits: train (-1), test (200); repo https://github.com/FlagOpen/TACO

## Metrics

- `pass@k` **(primary)** — range: [0, 1]
  - Measures the probability that at least one of k generated solutions passes all provided test cases. Computed as 1 - (C(n-k, c) / C(n, c)) where n is total attempts and c is correct attempts.

## Input / output format

**Input**: Natural language description of a programming problem, including constraints and test cases.

**Output**: Python 3 code block containing a complete program to solve the problem.

## Scoring recipe

```python
def pass_at_k(generated_programs, test_cases, k):
    passed = 0
    for prog in generated_programs[:k]:
        if run_all_tests(prog, test_cases):
            passed = 1
            break
    return passed
# Metric = mean(pass_at_k(prog_list, tests, k) for each problem)
```

## Common pitfalls

- Generation hyperparameters (temperature/top-p) must be tuned per difficulty level; fixed settings severely penalize harder problems.
- GPT-4 evaluation requires a specific prompt suffix and markdown code-block extraction, differing from open-weight model pipelines.
- Generated code undergoes a common truncation post-processing step before execution, which can silently drop valid solutions if not replicated.

## Evidence (verbatim from paper)

> Similar to the evaluations of the generation of standard code, for each programming problem, we allow the model to generate 200 attempts, using pass@k (k=1, 10, 100) as the evaluation metric.

## Citation

```bibtex
@misc{li2023taco,
  title={TACO: Topics in Algorithmic COde generation dataset},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2312.14852}
}
```

- arXiv: 2312.14852