Longbench Write Eval

---
name: longbench-write-eval
description: Evaluates an LLM's ability to generate ultra-long, coherent, and high-quality text (up to 20k words) while strictly adhering to explicit length constraints. It probes long-context generation capabilities, structural coherence over extended outputs, and instruction-following for length requirements. Use when the user wants to benchmark on LongBench-Write, or asks about evaluating this task. Reports Sq (Quality Score).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.07055
  bibtex_key: bai2024longwriter
  confidence: high
---

# longbench-write-eval

> LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs — Bai et al. (2024) (arXiv:2408.07055, 2024)

## What this evaluates

Evaluates an LLM's ability to generate ultra-long, coherent, and high-quality text (up to 20k words) while strictly adhering to explicit length constraints. It probes long-context generation capabilities, structural coherence over extended outputs, and instruction-following for length requirements.

## Datasets

- **LongBench-Write** — total ?; splits: test (-1); repo https://github.com/THUDM/LongWriter

## Metrics

- `Sq (Quality Score)` **(primary)** — range: [0, 100]
  - Overall quality score rated on a 0-100 scale, aggregating sub-dimensions (Relevance, Accuracy, Coherence, Clarity, Breadth and Depth, Reading Experience). Higher values indicate better generation quality.
- `Sl (Length Score)` — range: [0, 100]
  - Score measuring adherence to the required output length constraint, rated on a 0-100 scale. Evaluated separately across four output length brackets: [0, 500), [500, 2k), [2k, 4k), [4k, 20k).

## Input / output format

**Input**: A writing prompt specifying a topic and a required output length (e.g., 'Write a 20,000-word article about...').

**Output**: A single continuous text generation of arbitrary length.

## Scoring recipe

```python
# For each sample in LongBench-Write test set:
predictions = model.generate(prompt)
gold_length = extract_required_length(prompt)
actual_length = count_words(predictions)

# Quality & Length scores (0-100) are obtained via human/LLM annotation per sub-dimension
sq = compute_quality_score(predictions)
sl = compute_length_adherence_score(predictions, gold_length)

# Aggregate by length bracket
bracket = assign_bracket(actual_length)
record(sq, sl, actual_length, bracket)

# Final metric: mean Sq and Sl per bracket, plus overall mean
overall_sq = mean(sq for all samples)
overall_sl = mean(sl for all samples)
```

## Common pitfalls

- Models frequently truncate outputs well before the required length, especially beyond 4k words.
- Quality scores (Sq) often degrade significantly in longer brackets due to coherence loss, even when length is met.
- Length constraints in prompts are sometimes ignored or misinterpreted, leading to severe under-generation.

## Evidence (verbatim from paper)

> Table 7: Evaluation results on English samples in LongBench-Write.  

<table><tr><td rowspan="2"></td><td colspan="3">Overall</td><td colspan="2">[0, 500)</td><td colspan="2">[500, 2k)</td><td colspan="2">[2k, 4k)</td><td colspan="2">[4k, 20k)</td></tr><tr><td>S̅</td><td>Sl</td><td>Sq</td><td>Sl</td><td>Sq</td><td>Sl</td><td>Sq</td><td>Sl</td><td>Sq</td><td>Sl</td><td>Sq</td></tr>

## Citation

```bibtex
@misc{bai2024longwriter,
  title={LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2408.07055}
}
```

- arXiv: 2408.07055