Ege Math Assessment Eval

---
name: ege-math-assessment-eval
description: This benchmark evaluates vision-language models' ability to assess handwritten mathematical solutions against a standardized educational rubric. It probes the models' capacity for error diagnosis, step-by-step reasoning alignment, and accurate grade assignment under varying levels of contextual guidance. Use when the user wants to benchmark on EGE-Math Solutions Assessment Benchmark, or asks about evaluating this task. Reports final_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.22958
  bibtex_key: khrulev2025checkmat
  confidence: high
---

# ege-math-assessment-eval

> CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam — Khrulev et al. (2025) (arXiv:2507.22958, 2025)

## What this evaluates

This benchmark evaluates vision-language models' ability to assess handwritten mathematical solutions against a standardized educational rubric. It probes the models' capacity for error diagnosis, step-by-step reasoning alignment, and accurate grade assignment under varying levels of contextual guidance.

## Datasets

- **EGE-Math Solutions Assessment Benchmark** — total 122; splits: test (122); repo https://github.com/Karifannaa/Auto-check-EGE-math

## Metrics

- `final_score` **(primary)** — range: percent
  - Percentage of instances where the model's extracted final score exactly matches the expert-provided grade. Calculated as (number of exact matches / total instances) * 100.

## Input / output format

**Input**: A problem statement (text), a handwritten solution image, and optionally the correct final answer or a complete reference solution, depending on the evaluation mode.

**Output**: A structured assessment containing a step-by-step analysis of the solution, a final score, and a justification for that score.

## Scoring recipe

```python
def compute_final_score(predictions, gold_grades):
    correct = 0
    for pred, gold in zip(predictions, gold_grades):
        # Extract the numeric score from the model's structured output
        pred_score = extract_numeric_score(pred)
        if pred_score == gold:
            correct += 1
    return (correct / len(gold_grades)) * 100
```

## Common pitfalls

- Handwriting variability and low image resolution can cause recognition failures, leading to incorrect error diagnosis regardless of the model's mathematical reasoning.
- The rubric-based grading requires strict adherence to step-wise credit allocation; models often award partial credit incorrectly or ignore minor procedural deviations.
- Automated score extraction may fail if the model deviates from the exact prompt template format, causing parsing errors in the structured output.

## Evidence (verbatim from paper)

> The models were instructed to output their assessment in the structured format, including the analysis of the solution, the final score, and the justification for that score. This structured output facilitated automated extraction of the assigned scores for quantitative analysis.

## Citation

```bibtex
@misc{khrulev2025checkmat,
  title={CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam},
  author={Khrulev et al. (2025)},
  year={2025},
  note={arXiv:2507.22958}
}
```

- arXiv: 2507.22958