Spiqa Eval

---
name: spiqa-eval
description: Evaluates multimodal long-context reasoning and figure/table comprehension on scientific papers. Tests direct question answering with images, full paper context, and chain-of-thought retrieval capabilities. Use when the user wants to benchmark on SPIQA, or asks about evaluating this task. Reports L3Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.09413
  bibtex_key: pramanick2024spiqa
  confidence: high
---

# spiqa-eval

> SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers — Pramanick et al. (2024) (arXiv:2407.09413, 2024)

## What this evaluates

Evaluates multimodal long-context reasoning and figure/table comprehension on scientific papers. Tests direct question answering with images, full paper context, and chain-of-thought retrieval capabilities.

## Datasets

- **SPIQA** — total ?; splits: train (-1), test-A (-1), test-B (-1), test-C (-1); repo https://github.com/google/spiqa

## Metrics

- `L3Score` **(primary)** — range: other
  - A log-likelihood-based metric that computes token probabilities from an LLM to assess answer confidence and semantic equivalence, outperforming traditional token-matching metrics.
- `METEOR` — range: [0, 1]
  - Standard machine translation metric measuring alignment between generated and reference answers based on synonyms, stems, and exact matches.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores to measure consensus with reference answers.
- `ROUGE-L` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation measuring the longest common subsequence between generated and reference answers.
- `BERTScore F1` — range: [0, 1]
  - F1 score computed from contextual embeddings of generated and reference answers using a pre-trained BERT model.
- `Top-1 Retrieval Accuracy` — range: [0, 1]
  - Percentage of questions where the model correctly retrieves the single most helpful reference image.

## Input / output format

**Input**: Question text, reference images (figures/tables), and optionally full paper text. For open-weight models, images are provided one-by-one in a multi-turn setup.

**Output**: Free-form natural language answer. For CoT QA, step-by-step reasoning followed by the final answer, plus retrieval of reference images.

## Scoring recipe

```python
def evaluate(predictions, golds, questions, images, metric):
    scores = []
    for pred, gold, q, img in zip(predictions, golds, questions, images):
        if metric == 'L3Score':
            ll = compute_log_likelihood(pred, context=(q, img))
            scores.append(aggregate_ll(ll))
        elif metric == 'Top-1 Retrieval Accuracy':
            scores.append(1.0 if pred == gold else 0.0)
        else:
            scores.append(compute_standard_metric(pred, gold, metric))
    return mean(scores)
```

## Common pitfalls

- Omitting figure/table captions significantly drops performance for all models.
- Traditional metrics like ROUGE-L and BERTScore fail to correctly score semantically correct but lexically different answers.
- Models struggle with complex plots, charts, and tables requiring mathematical reasoning.

## Evidence (verbatim from paper)

> For evaluating the free-form answers, we report five different metrics for comprehensive analysis - METEOR, CIDEr, ROUGE-L, BERTScore F1 and the proposed L3Score. For the CoT QA task, we also report the top-1 accuracy for retrieving the helpful images to answer the question.

## Citation

```bibtex
@misc{pramanick2024spiqa,
  title={SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers},
  author={Pramanick et al. (2024)},
  year={2024},
  note={arXiv:2407.09413}
}
```

- arXiv: 2407.09413