Prismm Bench Eval

---
name: prismm-bench-eval
description: Evaluates large multimodal models' ability to detect, correct, and reason over real-world multimodal inconsistencies in scientific papers. It probes inter-modal mismatch detection, structured reasoning, and robustness to linguistic shortcuts versus genuine visual grounding. Use when the user wants to benchmark on PRISMM-Bench, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.16505
  bibtex_key: selch2025prismmbench
  confidence: high
---

# prismm-bench-eval

> PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies — Selch et al. (2025) (arXiv:2510.16505, 2025)

## What this evaluates

Evaluates large multimodal models' ability to detect, correct, and reason over real-world multimodal inconsistencies in scientific papers. It probes inter-modal mismatch detection, structured reasoning, and robustness to linguistic shortcuts versus genuine visual grounding.

## Datasets

- **PRISMM-Bench** — total 262; splits: test (262); repo https://github.com/da-luggas/prismm-bench

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly predicted options out of the total number of instances. Computed as (correct predictions / total instances) * 100.
- `Visual Reliance Ratio (R)` — range: percent
  - Measures dependence on visual context versus linguistic priors. Formula: R = (Acc_with_context - Acc_without_context) / (1 - Acc_without_context). Higher R indicates stronger reliance on visual grounding.

## Input / output format

**Input**: Multimodal scientific documents containing text, figures, tables, and equations. Inputs are provided at three contextual granularity levels: Focused (cropped images/text excerpts), Page, or Document (full PDF). Each instance presents a multiple-choice question to identify inconsistencies, propose remedies, or match pairs.

**Output**: Strictly the letter corresponding to the chosen option. For reasoning-enabled models, the final answer is extracted from within <think>...</think> tags before postprocessing.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_visual_reliance_ratio(acc_with_context, acc_without_context):
    return (acc_with_context - acc_without_context) / (1 - acc_without_context)
```

## Common pitfalls

- Models can exploit linguistic regularities in natural language multiple-choice options, achieving inflated accuracy without genuine multimodal reasoning.
- Context granularity significantly impacts performance; models degrade toward random chance when moving from Focused to Document context due to distraction and long-range grounding challenges.
- Reasoning traces must be explicitly parsed (e.g., extracting content between <think> tags) rather than assuming the final answer is always at the end of the generation.

## Evidence (verbatim from paper)

> Accuracy (%) of 21 diverse LMMs across three tasks (Ident, Remedy, Match) and three levels of contextual granularity (Sec.[3.4]). To quantify how much models and human rely on visual evidence versus linguistic priors, we compute the Visual Reliance Ratio R, adapted from the normalized Perceptual Score (Gat et al., [2021]): R = (Acc_with_context - Acc_without_context) / (1 - Acc_without_context)

## Citation

```bibtex
@misc{selch2025prismmbench,
  title={PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies},
  author={Selch et al. (2025)},
  year={2025},
  note={arXiv:2510.16505}
}
```

- arXiv: 2510.16505