Verisoftbench Eval

---
name: verisoftbench-eval
description: This benchmark probes an AI system's ability to perform repository-scale formal verification in Lean 4. It specifically tests context-aware proof automation, measuring how well models handle project-specific abstractions and transitive dependency closures beyond standard mathematical libraries. Use when the user wants to benchmark on VeriSoftBench-Full, or asks about evaluating this task. Reports solve_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.18307
  bibtex_key: [REDACTED]
  confidence: high
---

# verisoftbench-eval

> VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean — Xin et al. (2026) (arXiv:2602.18307, 2026)

## What this evaluates

This benchmark probes an AI system's ability to perform repository-scale formal verification in Lean 4. It specifically tests context-aware proof automation, measuring how well models handle project-specific abstractions and transitive dependency closures beyond standard mathematical libraries.

## Datasets

- **VeriSoftBench-Full** — total 500; splits: test (500); repo https://github.com/utopia-group/VeriSoftBench

## Metrics

- `solve_rate` **(primary)** — range: [0, 1]
  - Percentage of tasks marked as “Solved”. A task is solved if at least one of the k sampled candidate proof scripts, or any of their subsequent repair iterations, successfully passes Lean 4 verification.

## Input / output format

**Input**: Per instance: the target proof obligation/goal, the repository context (files, lemmas, dependencies), and for repair rounds: compiler error messages, relevant code lines, and the current proof state.

**Output**: A tactical proof block/script intended to satisfy the target goal.

## Scoring recipe

```python
solved_count = 0
for task in dataset:
    success = False
    for _ in range(k):
        script = model.generate(task.context, task.goal)
        if verify_lean4(script, task.repo_toolchain, timeout=300):
            success = True; break
        for _ in range(r):
            err = get_compiler_feedback(script)
            script = model.repair(script, err, task.context)
            if verify_lean4(script, task.repo_toolchain, timeout=300):
                success = True; break
        if success: break
    if success: solved_count += 1
return solved_count / len(dataset)
```

## Common pitfalls

- Must use the exact Lean 4 toolchain version associated with each source repository; pre-building environments and caching large dependencies like Mathlib is required for stable evaluation.
- Context presentation differs between VeriSoftBench-Full and VeriSoftBench-Aristotle; the latter requires a pre-compiled environment and includes same-file lemmas, which significantly impacts prover performance.
- Verification timeout is strictly 300 seconds per candidate script; exceeding this counts as a failure.

## Evidence (verbatim from paper)

> A task is marked as “Solved” if any of the $k$ candidates or their subsequent repairs pass verification. For each evaluation run, the model receives a task associated with the selected contextual configuration as input, and the evaluation proceeds in three stages: (1) we sample $k$ candidate proof scripts from the model. Each candidate is generated as a tactical proof block intended to satisfy the target goal; (2) we wrap each generated script into its original file context and attempt verification using the Lean 4 compiler. To ensure a stable and efficient evaluation, the all task environments are pre-built prior to verification; we use the specific toolchain version associated with the source repository of each task and enforce a 300-second compilation timeout per candidate

## Citation

```bibtex
@misc{xin2026verisoftbench,
  title={VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean},
  author={Xin et al. (2026)},
  year={2026},
  note={arXiv:2602.18307}
}
```

- arXiv: 2602.18307