Scitrek Eval

---
name: scitrek-eval
description: Evaluates long-context language models' ability to perform numerical aggregation, filtering, sorting, and logical operations across extended contexts (up to 1M tokens) using scientific article metadata and full-text articles. Use when the user wants to benchmark on SciTrek, or asks about evaluating this task. Reports exact match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.21028
  bibtex_key: li2025scitrek
  confidence: high
---

# scitrek-eval

> Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles — Li et al. (2025) (arXiv:2509.21028, 2025)

## What this evaluates

Evaluates long-context language models' ability to perform numerical aggregation, filtering, sorting, and logical operations across extended contexts (up to 1M tokens) using scientific article metadata and full-text articles.

## Datasets

- **SciTrek** — total ?; splits: train (19543), test (-1); repo https://github.com/oaimli/SciTrek

## Metrics

- `exact match` **(primary)** — range: [0, 1]
  - Binary score: 1 if the model's predicted answer exactly matches the ground truth string, 0 otherwise. Reported as the average across all instances.
- `F1` — range: [0, 1]
  - Token-level F1 score calculated between the predicted answer and the ground truth answer, averaged across instances.

## Input / output format

**Input**: Instruction prompt containing either the full-text scientific article or a corresponding database table as context, followed by a question generated via SQL queries over the article metadata.

**Output**: Three generated answers per question, expected to be factual items such as specific numbers, author names, or article titles.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = [token_f1(p, g) for p, g in zip(predictions, golds)]
    return {'exact_match': sum(em_scores) / len(em_scores), 'f1': sum(f1_scores) / len(f1_scores)}
```

## Common pitfalls

- The paper states models generate three answers per question but does not specify how to aggregate or select among them for the final metric calculation.
- Performance is highly sensitive to context length, with significant metric drops observed as input exceeds 128K tokens, making cross-length comparisons unstable.
- Out-of-distribution generalization (e.g., to unseen context lengths, topics, or skills like relational filtering) remains poor even after post-training, which can skew aggregate scores if not stratified.

## Evidence (verbatim from paper)

> We assess performance using average exact match and F1, as the expected outputs are factual items with minimal variation, such as specific numbers, author names, or article titles.

## Citation

```bibtex
@misc{li2025scitrek,
  title={Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2509.21028}
}
```

- arXiv: 2509.21028