Countqa Eval

---
name: countqa-eval
description: This benchmark evaluates the object counting and spatial individuation capabilities of multimodal large language models (MLLMs) on real-world images characterized by high density, clutter, and occlusion. It probes whether generalist models can perform precise, fine-grained visual grounding and numerical reasoning out-of-the-box without specialized training. Use when the user wants to benchmark on CountQA, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06585
  bibtex_key: [REDACTED]
  confidence: high
---

# countqa-eval

> CountQA: How Well Do MLLMs Count in the Wild? — Tamarapalli et al. (2025) (arXiv:2508.06585, 2025)

## What this evaluates

This benchmark evaluates the object counting and spatial individuation capabilities of multimodal large language models (MLLMs) on real-world images characterized by high density, clutter, and occlusion. It probes whether generalist models can perform precise, fine-grained visual grounding and numerical reasoning out-of-the-box without specialized training.

## Datasets

- **CountQA** — total 1528; splits: test (1528)

## Metrics

- `Exact Match (EM)` **(primary)** — range: percent
  - Percentage of predictions where the final extracted number is identical to the ground truth integer.
- `Relaxed Accuracy@5% (RA@5%)` — range: percent
  - Percentage of predictions where the predicted count falls within 5% of the ground truth value.
- `Relaxed Accuracy@10% (RA@10%)` — range: percent
  - Percentage of predictions where the predicted count falls within 10% of the ground truth value.

## Input / output format

**Input**: An image paired with a natural language question asking for the count of specific objects. Models are prompted with a system instruction to act as a counting assistant and return only a single integer.

**Output**: A single integer representing the predicted object count.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    em_count = 0
    ra5_count = 0
    ra10_count = 0
    for pred, gt in zip(predictions, ground_truths):
        if pred == gt:
            em_count += 1
        if abs(pred - gt) <= 0.05 * gt:
            ra5_count += 1
        if abs(pred - gt) <= 0.10 * gt:
            ra10_count += 1
    n = len(predictions)
    return {
        'Exact Match (EM)': em_count / n,
        'Relaxed Accuracy@5%': ra5_count / n,
        'Relaxed Accuracy@10%': ra10_count / n
    }
```

## Common pitfalls

- Models frequently fail to adhere to the strict single-integer output format, producing verbose text that requires external parsing (e.g., using an LLM) to extract the count.
- Counting performance degrades sharply as the number of objects increases, exposing a fundamental gap in numerical and spatial reasoning capabilities.
- High-density clutter and occlusion significantly impact accuracy, revealing deficits in fine-grained visual grounding.

## Evidence (verbatim from paper)

> The primary metric for our evaluation is Exact Match (EM), which measures the percentage of predictions where the final extracted number is identical to the ground truth integer. To provide a more nuanced view of performance, especially on questions with large counts, we also report on two Relaxed Accuracy (RA) metrics. An answer is considered correct under RA@5% or RA@10% if the predicted count falls within 5% or 10% of the ground truth value, respectively.

## Citation

```bibtex
@misc{tamarapalli2025countqa,
  title={CountQA: How Well Do MLLMs Count in the Wild?},
  author={Tamarapalli et al. (2025)},
  year={2025},
  note={arXiv:2508.06585}
}
```

- arXiv: 2508.06585