Livexiv Eval

---
name: livexiv-eval
description: This benchmark evaluates the multi-modal reasoning capabilities of Large Multimodal Models (LMMs) on scientific content scraped from ArXiv papers. It specifically probes visual question answering (VQA) on figures and table question answering (TQA) using multiple-choice formats derived from real-time academic publications. Use when the user wants to benchmark on LiveXiv, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.10783
  bibtex_key: [REDACTED]
  confidence: high
---

# livexiv-eval

> LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content — Shabtay et al. (2024) (arXiv:2410.10783, 2024)

## What this evaluates

This benchmark evaluates the multi-modal reasoning capabilities of Large Multimodal Models (LMMs) on scientific content scraped from ArXiv papers. It specifically probes visual question answering (VQA) on figures and table question answering (TQA) using multiple-choice formats derived from real-time academic publications.

## Datasets

- **LiveXiv** — total 16328; splits: test (16328); HF `LiveXiv/LiveXiv`; repo https://github.com/NimrodShabtay/LiveXiv

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted multiple-choice answers. Calculated as the number of instances where the model's predicted letter matches the ground truth letter divided by the total number of questions.

## Input / output format

**Input**: An image (figure or table) paired with a multiple-choice question and a set of answer options labeled with letters.

**Output**: A single letter corresponding to the chosen answer option.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Claude-Sonnet's reported accuracy may be inflated because it participated in the automated question-filtering pipeline, potentially biasing the dataset toward question types it is predisposed to solve.
- The benchmark is dynamic and versioned (v0-v4); results are not static and depend heavily on the specific release version and the generation/filtering models used.
- The efficient evaluation method (re-evaluating only 3-5 models to predict others) relies on Item Response Theory (IRT) and may yield higher prediction errors on smaller or more diverse dataset versions.

## Evidence (verbatim from paper)

> The output letter is then compared with the ground truth and the accuracy is measured. For ease of assimilation and to obtain insights into what type of data the models flourish at, we provide the results from data generated on tables and figures separately.

## Citation

```bibtex
@misc{shabtay2024livexiv,
  title={LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content},
  author={Shabtay et al. (2024)},
  year={2024},
  note={arXiv:2410.10783}
}
```

- arXiv: 2410.10783