Mqud Eval

---
name: mqud-eval
description: This benchmark evaluates whether vision-language models can generate scientifically grounded, figure-dependent questions rather than generic visual queries. It probes content-specific visual grounding by measuring how model outputs change when the correct figure is replaced, removed, or kept, alongside assessing the depth and diversity of the generated questions. Use when the user wants to benchmark on MQUD, or asks about evaluating this task. Reports rIG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23733
  bibtex_key: [REDACTED]
  confidence: high
---

# mqud-eval

> Multimodal QUD: Inquisitive Questions from Scientific Figures — Wu et al. (2026) (arXiv:2604.23733, 2026)

## What this evaluates

This benchmark evaluates whether vision-language models can generate scientifically grounded, figure-dependent questions rather than generic visual queries. It probes content-specific visual grounding by measuring how model outputs change when the correct figure is replaced, removed, or kept, alongside assessing the depth and diversity of the generated questions.

## Datasets

- **MQUD** — total 251; splits: test (200), analysis (51)

## Metrics

- `rIG` **(primary)** — range: other
  - Relative Information Gain. Computed as the normalized reduction in question negative log-likelihood (NLL) when conditioning on the correct figure versus text-only. Higher values indicate stronger figure dependency.
- `swap_gap` — range: other
  - Δ^swap_F = L_swap(Q_F) - L_to(Q_F), where L_swap is the NLL of the question generated with a wrong figure from the same paper, and L_to is the NLL with no figure. Positive values indicate content-specific grounding.
- `swap_pos_rate` — range: percent
  - Percentage of evaluation items where swap_gap > 0, indicating the model relies on specific figure content rather than generic visual cues.
- `question_quality` — range: percent
  - LLM-judged preference rates across dimensions (depth, figure specificity, diversity) via blind A/B testing against human expert ratings.

## Input / output format

**Input**: Title, abstract, figure image, and caption (trigger context). Surrounding paragraphs are provided for answer generation but questions are triggered only by the trigger context.

**Output**: A single natural language question ($Q_F$) grounded in the provided figure and text.

## Scoring recipe

```python
def compute_metrics(predictions, gold, model):
    nll_correct = [model.nll(q, fig) for q, fig in zip(predictions, gold['figures'])]
    nll_none = [model.nll(q, None) for q in predictions]
    nll_swap = [model.nll(q, gold['swap_figures']) for q in predictions]
    rIG = [(n - c) / n for n, c in zip(nll_none, nll_correct)]
    swap_gap = [s - n for s, n in zip(nll_swap, nll_none)]
    swap_pos_rate = sum(1 for g in swap_gap if g > 0) / len(swap_gap) * 100
    quality = llm_judge.evaluate(predictions, gold['questions'])
    return rIG, swap_gap, swap_pos_rate, quality
```

## Common pitfalls

- Confusing generic visual bias (where any figure lowers loss) with true content-specific grounding (where a wrong figure increases loss relative to no figure).
- Assuming text-only fine-tuning achieves equivalent visual grounding; ablation shows text-only SFT yields low rIG despite similar swap rates, proving the metrics capture distinct properties.
- Relying solely on proprietary models for evaluation, which prevents controlled diagnostics like training-dynamics tracking and swap-based ablations.

## Evidence (verbatim from paper)

> rIG increases from 0.60 [0.49, 0.73] to 0.97 [0.71, 1.25] ($p<0.0001$; Table[6.1](#S6.SS1 "6.1 Setup ‣ 6 Experiments ‣ Multimodal QUD: Inquisitive Questions from Scientific Figures")), and per-type analysis shows that figure-driven types retain more visual dependence than integration types (Appendix[B](#A2 "Appendix B Per-type visual information gain ‣ Multimodal QUD: Inquisitive Questions from Scientific Figures")). Notably, a text-only SFT ablation achieves comparable swap (76%) but much lower rIG (0.27), confirming that the two diagnostics capture distinct properties: swap reflects discourse-structural patterns, while rIG requires genuine visual grounding (Table[7](#A9.T7 "Table 7 ‣ Appendix I Additional results tables ‣ Multimodal QUD: Inquisitive Questions from Scientific Figures")).

## Citation

```bibtex
@misc{wu2026multimodalqud,
  title={Multimodal QUD: Inquisitive Questions from Scientific Figures},
  author={Wu et al. (2026)},
  year={2026},
  note={arXiv:2604.23733}
}
```

- arXiv: 2604.23733