Optbench Eval

---
name: optbench-eval
description: Evaluates formal theorem proving capabilities specifically within the undergraduate optimization domain. It probes a model's ability to generate syntactically correct and semantically progressive Lean 4 proof steps or full scripts under strict verifier constraints, while measuring robustness against catastrophic forgetting on general math benchmarks. Use when the user wants to benchmark on OptBench, MiniF2F-test, ProofNet-test, or asks about evaluating this task. Reports Pass@32.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23712
  bibtex_key: li2026optprover
  confidence: high
---

# optbench-eval

> OptProver: Bridging Olympiad and Optimization through Continual Training in Formal Theorem Proving — Li et al. (2026) (arXiv:2604.23712, 2026)

## What this evaluates

Evaluates formal theorem proving capabilities specifically within the undergraduate optimization domain. It probes a model's ability to generate syntactically correct and semantically progressive Lean 4 proof steps or full scripts under strict verifier constraints, while measuring robustness against catastrophic forgetting on general math benchmarks.

## Datasets

- **OptBench** — total 400; splits: test (400)
- **MiniF2F-test** — total ?; splits: test (-1)
- **ProofNet-test** — total ?; splits: test (-1)

## Metrics

- `Pass@1` — range: percent
  - Fraction of problems where the single generated proof attempt passes the Lean 4 verifier.
- `Pass@32` **(primary)** — range: percent
  - Fraction of problems where at least one of the 32 generated proof attempts passes the Lean 4 verifier.
- `Pass@256` — range: percent
  - Fraction of problems where at least one of the 256 generated proof attempts passes the Lean 4 verifier.

## Input / output format

**Input**: Current Lean 4 proof state and goal statement.

**Output**: A Lean 4 tactic or complete proof script.

## Scoring recipe

```python
def pass_at_k(proof_attempts, verifier, k):
    solved = 0
    for attempts in proof_attempts:
        if any(verifier(attempt) for attempt in attempts[:k]):
            solved += 1
    return (solved / len(proof_attempts)) * 100
```

## Common pitfalls

- Whole-proof and step-level provers use different evaluation budgets (Pass@32/256 vs Pass@1/32), making direct accuracy comparisons without context misleading.
- Syntactic correctness in Lean 4 does not guarantee proof progress; models often emit valid but stagnant tactics that fail to advance the goal.
- Pass@k measures the probability of finding at least one correct proof among k attempts, not the average number of correct proofs.

## Evidence (verbatim from paper)

> All whole-proof generation models are tested under Pass@32 and Pass@256. For step-level prover, we use an accumulative budget within Pass@1 and Pass@32.

## Citation

```bibtex
@misc{li2026optprover,
  title={OptProver: Bridging Olympiad and Optimization through Continual Training in Formal Theorem Proving},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.23712}
}
```

- arXiv: 2604.23712