Longform C Eval

---
name: longform-c-eval
description: Evaluates instruction-following long-form text generation capabilities of LLMs across diverse in-domain and out-of-domain tasks, including news summarization, recipe and story generation, long-form QA, and multilingual generation, while also measuring general language understanding via MMLU. Use when the user wants to benchmark on LongForm-C, Writing Prompts, ELI5, Recipe Generation, MMLU, MLSUM, or asks about evaluating this task. Reports METEOR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.08460
  bibtex_key: koksal2023longform
  confidence: high
---

# longform-c-eval

> LongForm: Effective Instruction Tuning with Reverse Instructions — Köksal et al. (2023) (arXiv:2304.08460, 2023)

## What this evaluates

Evaluates instruction-following long-form text generation capabilities of LLMs across diverse in-domain and out-of-domain tasks, including news summarization, recipe and story generation, long-form QA, and multilingual generation, while also measuring general language understanding via MMLU.

## Datasets

- **LongForm-C** — total ?; splits: test (-1)
- **Writing Prompts** — total 250; splits: test (250)
- **ELI5** — total 250; splits: test (250)
- **Recipe Generation** — total 250; splits: test (250)
- **MMLU** — total ?; splits: test (-1)
- **MLSUM** — total 400; splits: test (400)

## Metrics

- `METEOR` **(primary)** — range: [0, 1] (reported as 0–100 in tables)
  - Computes unigram precision and recall, applies a harmonic mean (F-mean), and penalizes fragmentation based on the longest common subsequence between reference and hypothesis.
- `BLEU` — range: [0, 1] (reported as 0–100)
  - Measures n-gram precision between reference and hypothesis with a brevity penalty for shorter outputs.
- `ROUGE` — range: [0, 1] (reported as 0–100)
  - Recall-oriented metric measuring overlap of n-grams, longest common subsequence, or summary words between reference and hypothesis.
- `self-BLEU` — range: [0, 1] (reported as 0–100)
  - Measures diversity within a set of generated hypotheses by computing BLEU scores between each pair of outputs.

## Input / output format

**Input**: Natural language instruction specifying the task, optionally with context/ingredients/title. For autoregressive models, an [EOI] token is appended between the instruction and the generation start to separate instruction from output.

**Output**: Generated long-form text response to the instruction.

## Scoring recipe

```python
def compute_metric(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        scores.append(meteor_score([ref], pred))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Relying on BLEU/ROUGE for long-form evaluation, as the paper explicitly notes they have limited correlation with human judgment for long texts.
- Assuming out-of-domain generalization datasets (WP, ELI5, Recipe) were part of the training data; they are explicitly held out to test generalization.
- Ignoring the [EOI] token requirement for autoregressive models, which is critical for correctly separating instruction from generation during inference.

## Evidence (verbatim from paper)

> As current metrics in text generation have limited capabilities in evaluating long text generation Celikyilmaz et al. ([2020]), we choose METEOR Banerjee and Lavie ([2005]) as our main metric as it exhibits higher human correlation Sharma et al. ([2017]); Chen et al. ([2022]). We evaluate them on the test set of LongForm-C... For generation, we perform nucleus sampling with p=0.9 for all LMs.

## Citation

```bibtex
@misc{koksal2023longform,
  title={LongForm: Effective Instruction Tuning with Reverse Instructions},
  author={Köksal et al. (2023)},
  year={2023},
  note={arXiv:2304.08460}
}
```

- arXiv: 2304.08460