Longalign Eval

---
name: longalign-eval
description: Evaluates large language models' ability to follow instructions and retrieve information in long-context scenarios (up to 64k tokens), while also measuring their general capabilities and instruction-following performance in short-context settings. Use when the user wants to benchmark on LongBench-Chat, LongBench, MT-Bench, ARC, HellaSwag, TruthfulQA, MMLU, or asks about evaluating this task. Reports GPT-4 rating (1-10).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.18058
  bibtex_key: [REDACTED]
  confidence: high
---

# longalign-eval

> LongAlign: A Recipe for Long Context Alignment of Large Language Models — Bai et al. (2024) (arXiv:2401.18058, 2024)

## What this evaluates

Evaluates large language models' ability to follow instructions and retrieve information in long-context scenarios (up to 64k tokens), while also measuring their general capabilities and instruction-following performance in short-context settings.

## Datasets

- **LongBench-Chat** — total ?; splits: test (-1)
- **LongBench** — total ?; splits: test (-1)
- **MT-Bench** — total ?; splits: test (-1)
- **ARC** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **TruthfulQA** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `GPT-4 rating (1-10)` **(primary)** — range: other
  - GPT-4 rates the model's output based on alignment with ground truth on a 1-10 scale. Scores are averaged across all test instances. For other datasets, standard metrics (accuracy, F1, ROUGE) are normalized to a 0-100 scale.

## Input / output format

**Input**: Instruction prompts paired with long or short context documents/texts.

**Output**: Model-generated text responses.

## Scoring recipe

```python
def compute_metric(predictions, gold, dataset_type):
    if dataset_type in ['LongBench-Chat', 'MT-Bench']:
        scores = [gpt4_rate(pred, gold) for pred, gold in zip(predictions, gold)]
        return sum(scores) / len(scores)
    else:
        raw = compute_standard_metric(predictions, gold)
        return normalize_0_100(raw)
```

## Common pitfalls

- Using ROUGE/F1 for LongBench instead of GPT-4, as the paper explicitly notes aligned models produce longer responses that penalize traditional metrics.
- Not averaging multiple GPT-4 ratings; the protocol specifies scoring twice and averaging for stability.
- Confusing LongBench-Chat (instruction-following) with LongBench (general QA/summarization), which use different evaluation focuses.

## Evidence (verbatim from paper)

> For LongBench-Chat and MT-Bench, the reported results are averaged over GPT-4's rating (1-10) across all test instances, while results on other datasets are normalized between 0-100.

## Citation

```bibtex
@misc{bai2024longalign,
  title={LongAlign: A Recipe for Long Context Alignment of Large Language Models},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2401.18058}
}
```

- arXiv: 2401.18058