Scivqa Eval

---
name: scivqa-eval
description: Evaluates multimodal LLMs on closed-ended visual and non-visual question answering over scientific figures. It probes recognition of visual attributes (color, shape, position) and reasoning capabilities across diverse chart types. Use when the user wants to benchmark on SciVQA, or asks about evaluating this task. Reports ROUGE-1 F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22141
  bibtex_key: abuhmad2025nfdi4ds
  confidence: high
---

# scivqa-eval

> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

## What this evaluates

Evaluates multimodal LLMs on closed-ended visual and non-visual question answering over scientific figures. It probes recognition of visual attributes (color, shape, position) and reasoning capabilities across diverse chart types.

## Datasets

- **SciVQA** — total 21000; splits: train (-1), test (-1); HF `katebor/SciVQA`; repo https://sdproc.org/2025/scivqa.html

## Metrics

- `ROUGE-1 F1` **(primary)** — range: [0, 1]
  - Unigram overlap F1 score between the predicted answer and the reference answer.

## Input / output format

**Input**: Image of a scientific figure, its caption, metadata, and a closed-ended question.

**Output**: Textual answer to the question.

## Scoring recipe

```python
pred_answer = model.generate(image, caption, metadata, question)
gold_answer = gold_label
rouge1_f1 = rouge1_score(gold_answer, pred_answer)
rougeL_f1 = rougeL_score(gold_answer, pred_answer)
bertscore_f1 = bertscore_f1(gold_answer, pred_answer)
```

## Common pitfalls

- The dataset mixes visual and non-visual questions, but evaluation is often reported as an aggregate score without separating them.
- Metrics include ROUGE-1, ROUGE-L, and BertScore, which measure different aspects of text similarity and may not align perfectly with factual correctness.

## Evidence (verbatim from paper)

> The evaluation is based on precision, recall and F1-scores of ROUGE-1, ROUGE-L, and BertScore. The highest-ranking team achieved scores of 0.80 for ROUGE-1 and ROUGE-L and 0.98 for BertScore.

## Citation

```bibtex
@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}
```

- arXiv: 2509.22141