Sci Verifybench Eval

---
name: sci-verifybench-eval
description: Evaluates a model's ability to perform cross-disciplinary scientific verification by judging the correctness or equivalence of proposed answers to scientific problems. It probes domain-specific logical reasoning, handling of complex mathematical/scientific transformations, and robustness to prompt variations. Use when the user wants to benchmark on SCI-VerifyBench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.24285
  bibtex_key: zheng2025sciverifier
  confidence: high
---

# sci-verifybench-eval

> SCI-Verifier: Scientific Verifier with Thinking — Zheng et al. (2025) (arXiv:2509.24285, 2025)

## What this evaluates

Evaluates a model's ability to perform cross-disciplinary scientific verification by judging the correctness or equivalence of proposed answers to scientific problems. It probes domain-specific logical reasoning, handling of complex mathematical/scientific transformations, and robustness to prompt variations.

## Datasets

- **SCI-VerifyBench** — total ?; splits: test (-1); repo https://github.com/Zhengsh123/SCI-Verifier

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly verified samples out of the total. Positive and negative samples are balanced by construction.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for the verification task, reported alongside Accuracy on VerifierBench and VerifyBench-Hard.

## Input / output format

**Input**: A scientific problem or statement paired with a proposed answer, often containing domain-specific equivalence transformations (e.g., formula rewrites, logical substitutions).

**Output**: A binary verification judgment (correct/incorrect) accompanied by chain-of-thought reasoning.

## Scoring recipe

```python
accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)
f1 = f1_score(golds, predictions, average='binary')
return accuracy, f1
```

## Common pitfalls

- Prompt sensitivity: general models lack an intrinsic notion of answer equivalence and rely heavily on prompt phrasing, causing performance drops when prompts are modified.
- Domain-dependent difficulty: Math and Physics tasks are significantly harder due to complex transformations (e.g., factorization, Taylor expansions), leading to lower scores compared to Biology/Chemistry/QA.
- Model scale is not decisive: Larger models do not consistently outperform smaller ones on verification tasks because the task relies more on prior knowledge and targeted reasoning than raw capacity.

## Evidence (verbatim from paper)

> For evaluation, we report Accuracy on SCI-VerifyBench, since positive and negative samples are balanced by construction. On VerifierBench and VerifyBench-hard, we additionally report F1 score alongside Accuracy. In all cases, higher values indicate stronger verification performance.

## Citation

```bibtex
@misc{zheng2025sciverifier,
  title={SCI-Verifier: Scientific Verifier with Thinking},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2509.24285}
}
```

- arXiv: 2509.24285