Murgat Attribution Eval

---
name: murgat-attribution-eval
description: Evaluates multimodal large language models' ability to generate verifiable, fact-level citations grounded in video and audio inputs. It probes whether models can correctly decompose reasoning into atomic claims and align them with precise temporal and modality-specific evidence without hallucinating references. Use when the user wants to benchmark on Video-MMMU, WorldSense, or asks about evaluating this task. Reports MURGAT-S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11509
  bibtex_key: wan2026multimodal
  confidence: high
---

# murgat-attribution-eval

> Multimodal Fact-Level Attribution for Verifiable Reasoning — Wan et al. (2026) (arXiv:2602.11509, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to generate verifiable, fact-level citations grounded in video and audio inputs. It probes whether models can correctly decompose reasoning into atomic claims and align them with precise temporal and modality-specific evidence without hallucinating references.

## Datasets

- **Video-MMMU** — total 100; splits: test (100)
- **WorldSense** — total 100; splits: test (100)

## Metrics

- `answer accuracy` — range: [0, 100]
  - String matching between the model's predicted answer choice and the golden answer choice.
- `MURGAT-S` **(primary)** — range: [0, 100]
  - Automated score combining coverage (fraction of claims with citations) and attribution quality (precision/recall of temporal and modality alignment). Normalized to 0-100. Correlates with human judgments at r=0.84.

## Input / output format

**Input**: Multimodal prompts containing video and/or audio inputs paired with questions. Models are evaluated under three prompting variants: direct generation (BASE), generation with inline citations (+CITATION), and post-hoc sentence-level citation generation (POST-HOC).

**Output**: Free-form reasoning text, a final answer choice, and optionally timestamped/modality-tagged citations for each claim or sentence.

## Scoring recipe

```python
def score_accuracy(pred, gold):
    return 1.0 if pred == gold else 0.0

def score_murgat_s(claims, citations):
    coverage = len(citations) / max(len(claims), 1)
    attribution = compute_alignment_precision_recall(claims, citations)
    return (coverage * attribution) * 100
```

## Common pitfalls

- High answer accuracy does not guarantee valid grounding; models frequently hallucinate citations while answering correctly.
- Post-hoc attribution methods may force-align abstract reasoning steps to random video segments, inflating coverage but destroying attribution quality.
- Vision-only models can generate audio citations despite lacking audio encoders, leading to misleadingly high grounding scores.

## Evidence (verbatim from paper)

> Overall, models struggle significantly with multimodal attribution, achieving a peak MURGAT-S of 69.2 on WorldSense and 56.9 on Video-MMMU (Gemini-3-Flash).

## Citation

```bibtex
@misc{wan2026multimodal,
  title={Multimodal Fact-Level Attribution for Verifiable Reasoning},
  author={Wan et al. (2026)},
  year={2026},
  note={arXiv:2602.11509}
}
```

- arXiv: 2602.11509