Scientific Figure Mcqa Eval

---
name: scientific-figure-mcqa-eval
description: Evaluates a model's ability to perform high-level visual reasoning and domain-specific knowledge grounding on scientific figures within a multiple-choice question answering setting. It specifically probes whether models can resist choice-induced prior bias where text-only answer options incorrectly steer predictions away from visually supported ground truth. Use when the user wants to benchmark on MAC, SciFIBench, MMSci, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28026
  bibtex_key: roh2026whenchoices
  confidence: high
---

# scientific-figure-mcqa-eval

> When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA — Roh et al. (2026) (arXiv:2603.28026, 2026)

## What this evaluates

Evaluates a model's ability to perform high-level visual reasoning and domain-specific knowledge grounding on scientific figures within a multiple-choice question answering setting. It specifically probes whether models can resist choice-induced prior bias where text-only answer options incorrectly steer predictions away from visually supported ground truth.

## Datasets

- **MAC** — total ?; splits: test (-1)
- **SciFIBench** — total ?; splits: test (-1)
- **MMSci** — total 3711; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1] or percent
  - Fraction of correctly predicted answer options out of the total number of instances.
- `Macro-F1` — range: [0, 1] or percent
  - Harmonic mean of precision and recall computed per answer class, then averaged across all classes to mitigate label imbalance.

## Input / output format

**Input**: A scientific figure (image) paired with a multiple-choice question and a fixed set of candidate answer options. Some benchmarks also include accompanying textual descriptions or context.

**Output**: A single selected answer option (e.g., A, B, C, or D) corresponding to the model's highest-scoring candidate.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_macro_f1(preds, gold, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- MMSci exhibits a highly imbalanced answer-label distribution and variable candidate counts, causing Accuracy and Macro-F1 to diverge when gains are concentrated on frequent labels.
- Contrastive decoding baselines (VCD, ICD) and the proposed method rely on hyperparameter coefficients (e.g., α=0.5) that weight the original vs. contrastive branches; default settings are used without per-dataset tuning.
- Models are prone to choice-induced prior bias, where semantically plausible text-only options are selected over visually correct ones, potentially inflating performance if not explicitly measured.

## Evidence (verbatim from paper)

> Performance is evaluated using Accuracy and Macro-F1 to account for potential class imbalance across benchmarks. We provide the formal definition of Macro-F1 in Appendix[A.1]. For VCD and ICD, we adopt the default hyperparameter settings from the original implementations without additional tuning. For SciCon, we use $\alpha\=0.5$ as the default setting throughout the main experiments.

## Citation

```bibtex
@misc{roh2026whenchoices,
  title={When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA},
  author={Roh et al. (2026)},
  year={2026},
  note={arXiv:2603.28026}
}
```

- arXiv: 2603.28026