Scigraphqa Eval

---
name: scigraphqa-eval
description: Evaluates multi-modal large language models' ability to interpret scientific graphs and generate accurate, context-aware answers in a multi-turn conversational setting. It probes open-vocabulary visual reasoning and the model's capacity to leverage auxiliary paper metadata for grounded responses. Use when the user wants to benchmark on SciGraphQA, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.03349
  bibtex_key: li2023scigraphqa
  confidence: high
---

# scigraphqa-eval

> SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs — Li et al. (2023) (arXiv:2308.03349, 2023)

## What this evaluates

Evaluates multi-modal large language models' ability to interpret scientific graphs and generate accurate, context-aware answers in a multi-turn conversational setting. It probes open-vocabulary visual reasoning and the model's capacity to leverage auxiliary paper metadata for grounded responses.

## Datasets

- **SciGraphQA** — total 295000; splits: test (3000)

## Metrics

- `CIDEr` **(primary)** — range: [0, 1]
  - Computes TF-IDF weighted n-gram similarity between predicted and reference answers, emphasizing rare but informative words to better capture consensus with human references.

## Input / output format

**Input**: Scientific graph image, contextual paper metadata (title, abstract, paragraph, graph text), and a multi-turn question.

**Output**: Textual answer string.

## Scoring recipe

```python
def compute_cider(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        pred_tokens = tokenize(pred)
        ref_tokens = tokenize(ref)
        score = cider_score(pred_tokens, ref_tokens)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Models often fail to maintain multi-turn context, answering only the latest question while ignoring prior turns.
- Injecting large amounts of paper metadata into prompts can overwhelm the model's attention mechanism, degrading graph-focused reasoning.
- BLEU and ROUGE may unfairly penalize technically correct but paraphrased scientific answers due to strict n-gram matching.

## Evidence (verbatim from paper)

> We set aside a test set of 3K samples and computed NLP metrics including BLEU-4, ROUGE, and CIDEr.

## Citation

```bibtex
@misc{li2023scigraphqa,
  title={SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.03349}
}
```

- arXiv: 2308.03349