Longgenbench Eval

---
name: longgenbench-eval
description: Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases. Use when the user wants to benchmark on LongGenBench-GSM8K, LongGenBench-MMLU, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.04199
  bibtex_key: liu2024longgenbench
  confidence: high
---

# longgenbench-eval

> LongGenBench: Long-context Generation Benchmark — Liu et al. (2024) (arXiv:2410.04199, 2024)

## What this evaluates

Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases.

## Datasets

- **LongGenBench-GSM8K** — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench
- **LongGenBench-MMLU** — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total K questions in the long-context prompt. Calculated as (correct_count / K) * 100.
- `delta` — range: percent
  - Performance degradation calculated as Baseline Accuracy minus LongGenBench Accuracy. Negative values indicate a drop in performance.

## Input / output format

**Input**: A long-context prompt containing K sequential questions from GSM8K or MMLU, requiring the model to generate a single continuous text response answering all questions.

**Output**: A single generated text response containing answers to all K questions in sequence.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, k):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / k) * 100
```

## Common pitfalls

- Confusing baseline accuracy (single-question evaluation) with LongGenBench accuracy (multi-question sequential generation).
- Assuming accuracy degradation is linear across question indices; the paper shows non-linear drops dependent on model architecture and size.
- Overlooking that the model must generate all K answers in a single pass without intermediate stopping or re-prompting.

## Evidence (verbatim from paper)

> Figure [3] shows the accuracy distribution of API accessed models in LongGenBench-GSM8K. The x-axis represents the question index within a single long-text response, with the maximum index being $K$. The y-axis indicates the accuracy of the model’s responses to these questions.

## Citation

```bibtex
@misc{liu2024longgenbench,
  title={LongGenBench: Long-context Generation Benchmark},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.04199}
}
```

- arXiv: 2410.04199