Pass At K

---
name: pass-at-k
description: Compute pass@k — the standard "any of N samples is correct" metric for code-generation evaluation (HumanEval / MBPP / LiveCodeBench / APPS / BigCodeBench / CodeContests). Use when the user has N samples per problem and wants the unbiased estimator of "probability at least one of the top-k is correct". Returns mean pass@k across the dataset, in [0, 1].
metadata:
  source_paper: arXiv:2107.03374 (Codex / HumanEval, Chen et al. 2021)
  task_type: code-eval-metric
  metrics: ["pass@k"]
  use_cases: [HumanEval, MBPP, APPS, LiveCodeBench, BigCodeBench, CodeContests]
---

# pass-at-k — Unbiased pass@k for code generation

Source: Chen et al., "Evaluating Large Language Models Trained on Code" (Codex paper, arXiv:2107.03374, §2.1). The unbiased estimator avoids the bias of the naive `1 - (1 - p)^k` plug-in when you only sampled a finite n.

## When to invoke this skill

- User has, per problem: `n` total samples, `c` of which pass the unit tests, and wants `pass@k` for some `k ≤ n`
- User says "compute pass@k", "HumanEval-style scoring", "pass@1 / pass@10 / pass@100"
- User mentions code-gen benchmarks (HumanEval, MBPP, APPS, LiveCodeBench, BigCodeBench)

## Definition

For a single problem with `n` total samples, `c` correct samples, and target cutoff `k`:

```
pass@k = 1 - C(n - c, k) / C(n, k)        if n - c >= k
       = 1                                 if n - c < k
```

Numerically stable form (paper-standard):
```
pass@k = 1 - prod_{i=n-c+1..n} (1 - k / i)        # iff k <= n - c
```

Then average over all problems in the dataset. **Use the unbiased estimator above, not `1 - (1 - c/n)^k`.**

## Reference implementation

```python
import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k for one problem. n total samples, c correct, cutoff k."""
    if k > n:
        raise ValueError(f"k={k} > n={n}; need to sample at least k completions")
    if n - c < k:
        return 1.0
    # 1 - prod_{i in [n-c+1 .. n]} (1 - k/i)
    return 1.0 - float(np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

def mean_pass_at_k(results: list[tuple[int, int]], k: int) -> float:
    """results = list of (n, c) per problem."""
    return float(np.mean([pass_at_k(n, c, k) for n, c in results]))

# example usage
# Per problem you sampled n=20 times and ran tests on each.
# results = [(20, 14), (20, 0), (20, 7), ...]
print(mean_pass_at_k(results, k=1))   # 0.35 — typical
print(mean_pass_at_k(results, k=10))  # higher; closer to coverage
```

The HuggingFace `evaluate` library has the same impl: `evaluate.load("code_eval")`.

## Input contract

- Per problem: integer `n` (samples drawn) and integer `c` (samples that passed all unit tests).
- `k`: cutoff. **Common reporting**: `pass@1` (greedy / temp 0.2 single sample), `pass@10`, `pass@100`. Must satisfy `k ≤ n`.
- Sampling: paper uses temperature 0.8 for `pass@10/100`, 0.2 for `pass@1`. Don't claim a high pass@k by sampling at temperature 0.

## Output format

```
HumanEval (164 problems, n=200, T=0.8):
  pass@1  = 0.282
  pass@10 = 0.461
  pass@100= 0.722
```

## Edge cases

- `c == 0`: contribution is exactly 0 for any k (no correct sample to draw).
- `c == n`: contribution is exactly 1 for any k ≤ n.
- `k > n`: undefined — your sampling protocol is incompatible. Either raise or downgrade k to n.
- Non-determinism in unit tests (timing flakes): re-run flaky tests, or wrap with a 60s timeout + fixed seeds. Do NOT count timeouts as `pass`.

## Don'ts

- Don't use `1 - (1 - c/n)^k` — biased upward when c is small.
- Don't compute pass@1 by sampling once and calling it pass@1 if `n=1` — variance is enormous; always sample n≥10 even for pass@1 and use the unbiased estimator.
- Don't rely on the model-output being valid Python — wrap unit-test execution in a sandbox + timeout. The paper warns about this explicitly.

## Citation

```bibtex
@article{chen2021codex,
  title={Evaluating Large Language Models Trained on Code},
  author={Chen, Mark and Tworek, Jerry and Jun, Heewoo and others},
  journal={arXiv:2107.03374}, year={2021}
}
```