Measurebench Eval

---
name: measurebench-eval
description: Evaluates vision-language models on fine-grained visual measurement reading, specifically testing their ability to accurately localize pointers and ticks on instrument scales, map visual cues to numerical values, and recognize measurement units from real-world and synthetic images. Use when the user wants to benchmark on MeasureBench, or asks about evaluating this task. Reports Overall accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26865
  bibtex_key: lin2025measurebench
  confidence: high
---

# measurebench-eval

> Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench — Lin et al. (2025) (arXiv:2510.26865, 2025)

## What this evaluates

Evaluates vision-language models on fine-grained visual measurement reading, specifically testing their ability to accurately localize pointers and ticks on instrument scales, map visual cues to numerical values, and recognize measurement units from real-world and synthetic images.

## Datasets

- **MeasureBench** — total ?; splits: real-world (-1), synthetic (-1)

## Metrics

- `Overall accuracy` **(primary)** — range: percent
  - Percentage of instances where both the predicted numerical value and unit exactly match the ground truth.
- `Value accuracy` — range: percent
  - Percentage of instances where the predicted numerical value exactly matches the ground truth value, regardless of unit prediction.
- `Unit accuracy` — range: percent
  - Percentage of instances where the predicted measurement unit exactly matches the ground truth unit.

## Input / output format

**Input**: An image of a measuring instrument (dial, digital, linear, or composite) accompanied by a prompt requesting the model to read the measurement.

**Output**: A textual response containing the predicted numerical value and the corresponding unit.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    correct_val = correct_unit = correct_overall = 0
    for pred, gold in zip(predictions, golds):
        p_val, p_unit = extract_value_and_unit(pred)
        g_val, g_unit = gold['value'], gold['unit']
        if p_val == g_val: correct_val += 1
        if p_unit == g_unit: correct_unit += 1
        if p_val == g_val and p_unit == g_unit: correct_overall += 1
    n = len(golds)
    return {
        'value_accuracy': correct_val / n * 100,
        'unit_accuracy': correct_unit / n * 100,
        'overall_accuracy': correct_overall / n * 100
    }
```

## Common pitfalls

- Error cancellation: models may arrive at the correct final number through flawed intermediate reasoning, inflating accuracy if only the final answer is scored.
- Unit recognition is near-ceiling (>90%), so overall accuracy is heavily driven by value reading difficulty, masking OCR capabilities.
- Synthetic vs real-world performance gap is small, so synthetic data may not fully capture real-world visual clutter/distortion challenges.

## Evidence (verbatim from paper)

> Table [2] reports results on MeasureBench for 17 VLMs. The best model, Gemini 2.5 Pro, reaches only 30.3% overall accuracy on real images and 26.1% on synthetic images, showing that reading measuring instruments remains a challenging fine-grained vision task for current VLMs.

## Citation

```bibtex
@misc{lin2025measurebench,
  title={Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2510.26865}
}
```

- arXiv: 2510.26865