Academiceval Eval

---
name: academiceval-eval
description: Evaluates an LLM's ability to perform long-context summarization and synthesize related work sections by retrieving and reasoning over heterogeneous academic memory chunks. Use when the user wants to benchmark on AcademicEval-abstract, AcademicEval-related, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12231
  bibtex_key: feng2026thoughtretriever
  confidence: medium
---

# academiceval-eval

> Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems — Tao Feng et al. (2026) (arXiv:2604.12231, 2026)

## What this evaluates

Evaluates an LLM's ability to perform long-context summarization and synthesize related work sections by retrieving and reasoning over heterogeneous academic memory chunks.

## Datasets

- **AcademicEval-abstract** — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/Thought-Retriever
- **AcademicEval-related** — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/Thought-Retriever

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Token-level F1 score computed between the model's generated text and the gold reference (original abstract, expert LLM summary, or original related work).
- `Win rate` — range: percent
  - Pairwise comparison metric indicating the percentage of instances where the model's output is preferred over a baseline's output.

## Input / output format

**Input**: For Abstract-single/multi: academic paper text with abstract and conclusion sections removed. For Related-multi: target paper title and abstract, plus a set of memory chunks containing abstracts of other papers (some cited, some random).

**Output**: Generated abstract (for single/multi tasks) or generated related work section (for related task).

## Scoring recipe

```python
def score(generated, gold):
    gen_tokens = set(generated.split())
    gold_tokens = set(gold.split())
    precision = len(gen_tokens & gold_tokens) / len(gen_tokens) if gen_tokens else 0
    recall = len(gen_tokens & gold_tokens) / len(gold_tokens) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- The gold label for the multi-paper abstract task is generated by an expert LLM rather than human annotators, which may propagate model biases.
- Evaluation conflates retrieval quality and generation quality in the Related-multi task, making it hard to isolate whether failures stem from chunk selection or text synthesis.

## Evidence (verbatim from paper)

> Experiments on AcademicEval show a 7.6% F1 gain and 16% win rate over baselines... The LLM is presented with one or more papers with the abstract and conclusion sections removed and is tasked with writing an abstract. For Abstract-single, the generated abstract is directly compared with the paper’s original abstract. For Abstract-multi, the generated abstract is compared with a summary of abstracts from all the provided papers, which is generated by an expert LLM as a label.

## Citation

```bibtex
@misc{feng2026thoughtretriever,
  title={Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems},
  author={Tao Feng et al. (2026)},
  year={2026},
  note={arXiv:2604.12231}
}
```

- arXiv: 2604.12231