Formationeval Eval

---
name: formationeval-eval
description: Evaluates large language models' domain knowledge in petroleum geoscience using a 505-question multiple-choice benchmark. It probes understanding across seven specialized subdomains, including petrophysics, reservoir engineering, and drilling, while measuring performance variance by model size, cost, and question difficulty. Use when the user wants to benchmark on FormationEval, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02158
  bibtex_key: ermilov2026formationeval
  confidence: high
---

# formationeval-eval

> FormationEval, an open multiple-choice benchmark for petroleum geoscience — Ermilov (2026) (arXiv:2601.02158, 2026)

## What this evaluates

Evaluates large language models' domain knowledge in petroleum geoscience using a 505-question multiple-choice benchmark. It probes understanding across seven specialized subdomains, including petrophysics, reservoir engineering, and drilling, while measuring performance variance by model size, cost, and question difficulty.

## Datasets

- **FormationEval** — total 505; splits: test (505); repo https://github.com/AlmazErmilov/FormationEval-an-Open-Benchmark-for-Oil-Gas-Geoscience-MCQ-Evaluation

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total 505 items.

## Input / output format

**Input**: Multiple-choice question with four options (A, B, C, D) covering one of seven petroleum geoscience domains.

**Output**: The model's selected option letter (e.g., 'A', 'B', 'C', or 'D').

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Residual length bias: correct answers are disproportionately the longest option, especially in hard questions (55.6% of the time).
- Medium questions have higher calculation rates (8.8%) and are more comparison-heavy, making them harder than expected relative to difficulty labels.
- Petrophysics domain consistently yields 3–5 percentage points lower accuracy due to technical well-logging concepts.

## Evidence (verbatim from paper)

> Accuracy spans a wide range, from 99.8% (Gemini 3 Pro Preview) to 57.6% (Llama-3.2-3b-instruct).

## Citation

```bibtex
@misc{ermilov2026formationeval,
  title={FormationEval, an open multiple-choice benchmark for petroleum geoscience},
  author={Ermilov (2026)},
  year={2026},
  note={arXiv:2601.02158}
}
```

- arXiv: 2601.02158