Nim Benchmark Eval

---
name: nim-benchmark-eval
description: Evaluates multimodal LLMs' ability to locate and reason about fine-grained details in complex real-world documents. It specifically probes resilience against irrelevant information (distractor images) and measures performance across open- and closed-domain retrieval settings. Use when the user wants to benchmark on ArxiVQA, DUDE, NiM-Benchmark, or asks about evaluating this task. Reports Exact-Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05053
  bibtex_key: [REDACTED]
  confidence: high
---

# nim-benchmark-eval

> Finding Needles in Images: Can Multimodal LLMs Locate Fine Details? — Parth Thakkar et al. (arXiv:2508.05053, 2025)

## What this evaluates

Evaluates multimodal LLMs' ability to locate and reason about fine-grained details in complex real-world documents. It specifically probes resilience against irrelevant information (distractor images) and measures performance across open- and closed-domain retrieval settings.

## Datasets

- **ArxiVQA** — total ?; splits: test (500)
- **DUDE** — total ?; splits: test (500)
- **NiM-Benchmark** — total 937; splits: test (937)

## Metrics

- `Exact-Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer string exactly matches the gold answer string, 0 otherwise.
- `F1-Score` — range: [0, 1]
  - Token-level F1 score computed as the harmonic mean of precision and recall between predicted and gold answers.
- `ANLS Score` — range: [0, 1]
  - Average Normalized Levenshtein Similarity; computes character-level overlap normalized by the length of the gold answer.
- `Accuracy` — range: [0, 1]
  - Fraction of correctly answered multiple-choice questions, used specifically for ArxiVQA.

## Input / output format

**Input**: Question text and one or more context images. In open-domain settings, top-k retrieved images are provided; in closed-domain, a predefined set containing the exact context. NiM-Benchmark additionally includes distractor images.

**Output**: Textual answer string generated by the model.

## Scoring recipe

```python
def compute_metrics(predictions, golds, dataset_name):
    em = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1 = [token_f1(p, g) for p, g in zip(predictions, golds)]
    anls = [char_levenshtein_sim(p, g) for p, g in zip(predictions, golds)]
    if dataset_name == 'ArxiVQA':
        return {'accuracy': sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)}
    return {'EM': np.mean(em), 'F1': np.mean(f1), 'ANLS': np.mean(anls)}
```

## Common pitfalls

- ArxiVQA evaluation uses the training set because the official test split is unavailable, which may skew results compared to standard benchmarks.
- Hyperparameters are tuned on a small random subset of 50 questions per dataset, potentially leading to overfitting to the tuning set.
- NiM-Benchmark includes distractor images to test resilience; many baseline models fail to filter irrelevant context, causing performance drops.

## Evidence (verbatim from paper)

> We use Exact-Match (EM), F1-Score (Rajpurkar, 2016), and ANLS Score (Biten et al., 2019) as automatic metrics to assess the correctness of the predicted answers. For ArxiVQA, being a multiple-choice question dataset, we use accuracy as the evaluation metric.

## Citation

```bibtex
@misc{thakkar2025findingneedles,
  title={Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?},
  author={Parth Thakkar et al.},
  year={2025},
  note={arXiv:2508.05053}
}
```

- arXiv: 2508.05053