Slideagent Eval

---
name: slideagent-eval
description: Evaluates multi-page visual document understanding and question answering. It probes a model's ability to retrieve relevant slides, perform spatial and layout reasoning, and accurately extract numeric values or generate lexical matches for open-ended answers. Use when the user wants to benchmark on SlideVQA, TechSlides, FinSlides, or asks about evaluating this task. Reports Num, Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26615
  bibtex_key: jin2025slideagent
  confidence: high
---

# slideagent-eval

> SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding — Jin et al. (2025) (arXiv:2510.26615, 2025)

## What this evaluates

Evaluates multi-page visual document understanding and question answering. It probes a model's ability to retrieve relevant slides, perform spatial and layout reasoning, and accurately extract numeric values or generate lexical matches for open-ended answers.

## Datasets

- **SlideVQA** — total ?; splits: test (-1)
- **TechSlides** — total ?; splits: test (-1)
- **FinSlides** — total ?; splits: test (-1)

## Metrics

- `Num` **(primary)** — range: percent
  - Extracts numeric values from predictions and ground truth, normalizes formats (e.g., '17k' to '17000', '97%' to '0.97'), and checks for exact match.
- `F1` — range: [0, 1]
  - Computes token-level F1 score between predicted and ground-truth answers after normalization, tokenization, and removal of stopwords and punctuation.
- `Overall` **(primary)** — range: percent
  - Composite headline metric aggregating performance across numeric and lexical tasks, reported as a percentage score.

## Input / output format

**Input**: Multi-page slide images (or concatenated top-3 retrieved images for single-image models) and a natural language query. Optionally, ground-truth page indices are provided in the oracle setting.

**Output**: A natural language text answer.

## Scoring recipe

```python
def score(pred, gold):
    if is_numeric_query(gold):
        pred_norm = normalize_number(pred)
        gold_norm = normalize_number(gold)
        return 1.0 if pred_norm == gold_norm else 0.0
    else:
        pred_tokens = preprocess(pred) # remove stopwords/punct, tokenize
        gold_tokens = preprocess(gold)
        return f1_score(pred_tokens, gold_tokens)
```

## Common pitfalls

- Number normalization must handle diverse formats (percentages, decimals, word-based like 'thousand') consistently before comparison.
- The 'Overall' metric is a composite score; readers should not assume it equals simple arithmetic mean of Num and F1 without checking the paper's exact weighting.
- Retrieval noise significantly impacts end-to-end scores; oracle settings (ground-truth pages provided) isolate reasoning capability and yield higher scores.

## Evidence (verbatim from paper)

> For questions asking about numeric values, we extract, standardize, and compare the prediction and the ground-truth in various formats, including percentages, decimals, integers, and word-based representations (e.g., “three”, “thousand”, “million”). Numbers are normalized to a unified format (e.g., ‘17k’ $
ightarrow$ ‘17000’, ‘2.5 million’ $
ightarrow$ ‘2500000’, ‘97%’ $
ightarrow$ ‘0.97). Otherwise, we use F1-score to evaluate the lexical overlap between predicted and ground-truth answers. Both answers are normalized, tokenized, and preprocessed by removing stopwords and punctuation before metric calculation.

## Citation

```bibtex
@misc{jin2025slideagent,
  title={SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding},
  author={Jin et al. (2025)},
  year={2025},
  note={arXiv:2510.26615}
}
```

- arXiv: 2510.26615