Scifibench Eval

---
name: scifibench-eval
description: This benchmark evaluates large multimodal models' ability to interpret scientific figures by testing their capacity to match figures to captions and vice versa. It probes fine-grained visual-textual reasoning, attention to scientific details, and robustness against adversarially selected distractors. Use when the user wants to benchmark on SciFIBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.08807
  bibtex_key: roberts2024scifibench
  confidence: high
---

# scifibench-eval

> SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation — Roberts et al. (2024) (arXiv:2405.08807, 2024)

## What this evaluates

This benchmark evaluates large multimodal models' ability to interpret scientific figures by testing their capacity to match figures to captions and vice versa. It probes fine-grained visual-textual reasoning, attention to scientific details, and robustness against adversarially selected distractors.

## Datasets

- **SciFIBench** — total 2000; splits: test (2000); repo https://github.com/jonathan-roberts1/SciFIBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as the number of instances where the model's predicted choice letter matches the ground truth, divided by the total number of instances.

## Input / output format

**Input**: For Figure→Caption: a scientific figure image and four multiple-choice caption options. For Caption→Figure: a figure caption and four multiple-choice figure options. Both are provided with a 0-shot chain-of-thought prompt template.

**Output**: A single choice letter (e.g., 'A', 'B', 'C', or 'D') corresponding to the correct option. Models are instructed to output only the letter, though they often generate verbose reasoning.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_letters):
    correct = 0
    for pred, gold in zip(predictions, gold_letters):
        extracted = extract_letter(pred) # via regex or LLM parser
        if extracted == gold:
            correct += 1
    return (correct / len(gold_letters)) * 100
```

## Common pitfalls

- Models frequently ignore the instruction to output only a choice letter, requiring post-hoc parsing (e.g., via an LLM) to extract the answer.
- Performance is highly sensitive to the quality of multiple-choice distractors; adversarial negatives significantly lower accuracy compared to randomly selected ones.
- Shuffling the order of multiple-choice options can cause performance to fluctuate by up to 5%.

## Evidence (verbatim from paper)

> As expected, in the vast majority of cases, accuracy scores are higher on the random negatives – for some open-source models, the accuracy score more than doubles, and for the closed-source models, the maximum accuracy score is almost met.

## Citation

```bibtex
@misc{roberts2024scifibench,
  title={SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation},
  author={Roberts et al. (2024)},
  year={2024},
  note={arXiv:2405.08807}
}
```

- arXiv: 2405.08807