Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases. Use when the user wants to benchmark on LongGenBench-GSM8K, LongGenBench-MMLU, or asks about evaluating this task. Reports accuracy.
Installs into .claude/skills of the current project.
Are you the author of Longgenbench Eval?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-longgenbench-eval)
---
name: longgenbench-eval
description: Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases. Use when the user wants to benchmark on LongGenBench-GSM8K, LongGenBench-MMLU, or asks about evaluating this task. Reports accuracy.
metadata:
skill_kind: dataset_eval
source_arxiv: 2410.04199
bibtex_key: [REDACTED]
confidence: high
---
# longgenbench-eval
> LongGenBench: Long-context Generation Benchmark — Liu et al. (2024) (arXiv:2410.04199, 2024)
## What this evaluates
Evaluates the ability of LLMs to maintain accuracy and logical consistency when generating long-text responses that answer multiple sequential questions from GSM8K or MMLU in a single pass. It specifically probes performance degradation as the number of generated questions increases.
## Datasets
- **LongGenBench-GSM8K** — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench
- **LongGenBench-MMLU** — total ?; splits: test (-1); repo https://github.com/Dominic789654/LongGenBench
## Metrics
- `accuracy` **(primary)** — range: percent
- Percentage of correctly answered questions out of the total K questions in the long-context prompt. Calculated as (correct_count / K) * 100.
- `delta` — range: percent
- Performance degradation calculated as Baseline Accuracy minus LongGenBench Accuracy. Negative values indicate a drop in performance.
## Input / output format
**Input**: A long-context prompt containing K sequential questions from GSM8K or MMLU, requiring the model to generate a single continuous text response answering all questions.
**Output**: A single generated text response containing answers to all K questions in sequence.
## Scoring recipe
```python
def compute_accuracy(predictions, gold, k):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return (correct / k) * 100
```
## Common pitfalls
- Confusing baseline accuracy (single-question evaluation) with LongGenBench accuracy (multi-question sequential generation).
- Assuming accuracy degradation is linear across question indices; the paper shows non-linear drops dependent on model architecture and size.
- Overlooking that the model must generate all K answers in a single pass without intermediate stopping or re-prompting.
## Evidence (verbatim from paper)
> Figure [3] shows the accuracy distribution of API accessed models in LongGenBench-GSM8K. The x-axis represents the question index within a single long-text response, with the maximum index being $K$. The y-axis indicates the accuracy of the model’s responses to these questions.
## Citation
```bibtex
@misc{liu2024longgenbench,
title={LongGenBench: Long-context Generation Benchmark},
author={Liu et al. (2024)},
year={2024},
note={arXiv:2410.04199}
}
```
- arXiv: 2410.04199
