Musciclaims Eval

---
name: musciclaims-eval
description: Multimodal scientific claim verification, requiring models to read complex figures and captions to determine if a scientific claim is supported, neutral, or contradicted. It also probes evidence localization, basic visual understanding, cross-modal aggregation, and epistemic sensitivity. Use when the user wants to benchmark on MuSciClaims, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04585
  bibtex_key: lal2025musciclaims
  confidence: high
---

# musciclaims-eval

> MuSciClaims: Multimodal Scientific Claim Verification — Lal et al. (2025) (arXiv:2506.04585, 2025)

## What this evaluates

Multimodal scientific claim verification, requiring models to read complex figures and captions to determine if a scientific claim is supported, neutral, or contradicted. It also probes evidence localization, basic visual understanding, cross-modal aggregation, and epistemic sensitivity.

## Datasets

- **MuSciClaims** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed per class and macro-averaged across Support, Neutral, and Contradict.
- `Exact Match` — range: [0, 1]
  - Binary metric returning 1 if the model's one-word answer exactly matches the gold answer, else 0.

## Input / output format

**Input**: Multi-panel figure, figure caption, and a scientific claim.

**Output**: For ClaimVerification: a prediction of 'Support', 'Neutral', or 'Contradict'. For EvidenceLocalization: relevant panel names plus the prediction. For BasicVisualUnderstanding: a single-word answer.

## Scoring recipe

```python
def compute_f1(predictions, golds):
    tp = fp = fn = 0
    for p, g in zip(predictions, golds):
        if p == g: tp += 1
        elif p != g: fp += 1; fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_em(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
```

## Common pitfalls

- Models exhibit a strong bias toward predicting 'Support' regardless of the actual evidence.
- Models frequently fail to correctly localize evidence to specific figure panels or aggregate cross-modal information.
- Models show low epistemic sensitivity, often failing to change predictions consistently when claims are perturbed.

## Evidence (verbatim from paper)

> MuSciClaims is designed as a ClaimVerification task. Each data point contains a claim, an associated (multi-panel) figure (and caption) and a label (Support, Neutral, Contradict). Given the figure (and caption) and a claim, models must generate a prediction about whether the claim is supported. We evaluate models on this task using standard metrics of precision, recall and F1 score.

## Citation

```bibtex
@misc{lal2025musciclaims,
  title={MuSciClaims: Multimodal Scientific Claim Verification},
  author={Lal et al. (2025)},
  year={2025},
  note={arXiv:2506.04585}
}
```

- arXiv: 2506.04585