Scimdr Eval

---
name: scimdr-eval
description: Evaluates a model's ability to perform complex, claim-centric reasoning over full scientific documents containing multimodal elements (charts, tables, figures). It probes the model's capacity to localize evidence and answer questions accurately despite long-context noise and distractors. Use when the user wants to benchmark on SciMDR-Eval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.12249
  bibtex_key: chen2026scimdr
  confidence: high
---

# scimdr-eval

> SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning — Chen et al. (2026) (arXiv:2603.12249, 2026)

## What this evaluates

Evaluates a model's ability to perform complex, claim-centric reasoning over full scientific documents containing multimodal elements (charts, tables, figures). It probes the model's capacity to localize evidence and answer questions accurately despite long-context noise and distractors.

## Datasets

- **SciMDR-Eval** — total 300000; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions on the test set, computed as exact-match or semantic match depending on the baseline implementation.

## Input / output format

**Input**: Full scientific research papers (text and embedded figures/tables) paired with claim-centric questions requiring multi-step reasoning.

**Output**: Natural language text answers to the posed questions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Long-context noise causes severe attention dilution; performance drops significantly when full papers are provided vs. oracle context.
- Simple QA pairs without explicit reasoning chains fail to teach complex scientific logic, causing massive accuracy drops.
- Synthetic data generation must balance faithfulness to source claims with realism of full-document context to avoid training degradation.

## Evidence (verbatim from paper)

> Table 7: Challenge of Attention Dilution. Effect of context noise on SciMDR-Eval accuracy. Performance degrades as the amount of irrelevant context increases.

## Citation

```bibtex
@misc{chen2026scimdr,
  title={SciMDR: Benchmarking and Advancing Scientific Multimodal Document Reasoning},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.12249}
}
```

- arXiv: 2603.12249