Scizoom Eval

---
name: scizoom-eval
description: Evaluates a model's ability to perform hierarchical scientific summarization by generating three distinct granularity levels (Abstract, Key Contributions, TL;DR) from a single full-text input. It probes multi-granularity text compression and the model's capacity to maintain coherence across varying compression ratios within a single inference pass. Use when the user wants to benchmark on SciZoom, or asks about evaluating this task. Reports unspecified summarization metric.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.16131
  bibtex_key: jang2026scizoom
  confidence: medium
---

# scizoom-eval

> SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era — Jang et al. (2026) (arXiv:2603.16131, 2026)

## What this evaluates

Evaluates a model's ability to perform hierarchical scientific summarization by generating three distinct granularity levels (Abstract, Key Contributions, TL;DR) from a single full-text input. It probes multi-granularity text compression and the model's capacity to maintain coherence across varying compression ratios within a single inference pass.

## Datasets

- **SciZoom** — total 44946; splits: test (44946); repo https://github.com/janghana/SciZoom

## Metrics

- `unspecified summarization metric` **(primary)** — range: other
  - Not explicitly defined in the provided text. Standard automatic summarization metrics (e.g., ROUGE, BERTScore) are typically used for this task.

## Input / output format

**Input**: Full text of a scientific paper ($G_1$), with abstracts, bibliographies, and formatting artifacts removed.

**Output**: Three hierarchical summaries: Abstract ($G_2$), Key Contributions ($G_3$), and TL;DR ($G_4$).

## Scoring recipe

```python
def score(predictions, gold):
    scores = {}
    for level in ['abstract', 'contributions', 'tldr']:
        if level in gold and gold[level] is not None:
            scores[level] = compute_metric(predictions[level], gold[level])
        else:
            scores[level] = None
    return scores
```

## Common pitfalls

- TL;DR ground truth is only available for 21,295 papers (47.4% of the corpus), meaning evaluation coverage is incomplete for this granularity level.
- Key Contributions for 66% of papers were synthetically generated via an LLM fallback pipeline, which may introduce noise or bias into the gold standard.
- The Pre/Post-LLM temporal split is based on submission deadlines (e.g., ICLR 2023 is Pre-LLM), not publication dates, which can cause temporal misalignment in era-based analysis.

## Evidence (verbatim from paper)

> For the TL;DR evaluation, we use the subset of 21,295 papers with author-provided ground truth, covering 47.4% of the corpus.

## Citation

```bibtex
@misc{jang2026scizoom,
  title={SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2603.16131}
}
```

- arXiv: 2603.16131