M2 Verify Eval

---
name: m2-verify-eval
description: Evaluates a model's ability to verify scientific claims by cross-referencing textual assertions with provided multimodal evidence (figures/diagrams). It probes cross-modal reasoning, spatial/anatomical understanding, and the generation of factually grounded explanations. Use when the user wants to benchmark on M2-Verify-Med, M2-Verify-Gen, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.01306
  bibtex_key: ansari2026m2verify
  confidence: high
---

# m2-verify-eval

> M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency — Ansari et al. (2026) (arXiv:2604.01306, 2026)

## What this evaluates

Evaluates a model's ability to verify scientific claims by cross-referencing textual assertions with provided multimodal evidence (figures/diagrams). It probes cross-modal reasoning, spatial/anatomical understanding, and the generation of factually grounded explanations.

## Datasets

- **M2-Verify-Med** — total ?; splits: test (-1)
- **M2-Verify-Gen** — total ?; splits: test (-1)

## Metrics

- `Macro-F1` **(primary)** — range: [0, 1]
  - The unweighted mean of recall or precision for each class. Calculated as the average of per-class F1 scores across all verification labels.
- `Micro-F1` — range: [0, 1]
  - The global F1 score calculated by summing true positives, false positives, and false negatives across all classes before computing the metric.
- `BLEU-2` — range: [0, 1]
  - N-gram precision metric measuring the overlap of 2-grams between generated explanations and gold references.
- `ROUGE-L` — range: [0, 1]
  - Recall-oriented metric based on the longest common subsequence between generated and reference explanations.
- `METEOR` — range: [0, 1]
  - N-gram metric that incorporates synonymy, stemming, and paraphrasing to measure alignment between generated and reference explanations.
- `LLM-as-a-Judge` — range: [0, 10]
  - Automated scoring of explanation quality across five dimensions: correctness, relevance, completeness, entailment, and clarity, using a dedicated LLM judge.
- `Expert Faithfulness` — range: [1, 5]
  - Human expert rating on a 1-5 scale assessing the logical faithfulness and factual accuracy of model-generated explanations.

## Input / output format

**Input**: A textual scientific claim paired with a corresponding scientific figure or diagram (image).

**Output**: A verification label (e.g., support/refute) and a generated natural language explanation justifying the claim-evidence alignment.

## Scoring recipe

```python
def evaluate(predictions, gold):
    labels_pred = [p['label'] for p in predictions]
    labels_gold = [g['label'] for g in gold]
    macro_f1 = f1_score(labels_gold, labels_pred, average='macro')
    micro_f1 = f1_score(labels_gold, labels_pred, average='micro')
    exp_pred = [p['explanation'] for p in predictions]
    exp_gold = [g['explanation'] for g in gold]
    bleu2 = compute_bleu(exp_gold, exp_pred, n=2)
    rouge_l = compute_rouge(exp_gold, exp_pred, type='L')
    meteor = compute_meteor(exp_gold, exp_pred)
    return {'macro_f1': macro_f1, 'micro_f1': micro_f1, 'bleu2': bleu2, 'rouge_l': rouge_l, 'meteor': meteor}
```

## Common pitfalls

- Models frequently exhibit 'visual bypassing', ignoring provided images and relying on parametric memory instead.
- Performance degrades sharply on high-complexity perturbations (e.g., anatomical shifts) compared to low-complexity ones, which can be misinterpreted as general capability failure rather than reasoning complexity limits.
- LLM-as-a-Judge scores for semantic validity often diverge from human expert faithfulness ratings, requiring careful calibration.

## Evidence (verbatim from paper)

> Following (Zhang and Lee, 2025) we report Macro F1 and Micro F1 scores for verification. For explanations we calculate N-gram metrics (BLEU-2, ROUGE-L, and METEOR) alongside an LLM-as-a-Judge measuring correctness, relevance, completeness, entailment, and clarity.

## Citation

```bibtex
@misc{ansari2026m2verify,
  title={M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency},
  author={Ansari et al. (2026)},
  year={2026},
  note={arXiv:2604.01306}
}
```

- arXiv: 2604.01306