Longlamp Eval

---
name: longlamp-eval
description: Evaluates a model's ability to generate personalized long-form text by integrating retrieved user profiles into a retrieval-augmented generation framework. It probes how well models can adapt their writing style and content to specific user attributes across different domains like emails, abstracts, reviews, and topic-based writing. Use when the user wants to benchmark on LongLaMP, or asks about evaluating this task. Reports METEOR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.11016
  bibtex_key: kumar2024longlamp
  confidence: high
---

# longlamp-eval

> LongLaMP: A Benchmark for Personalized Long-form Text Generation — Kumar et al. (2024) (arXiv:2407.11016, 2024)

## What this evaluates

Evaluates a model's ability to generate personalized long-form text by integrating retrieved user profiles into a retrieval-augmented generation framework. It probes how well models can adapt their writing style and content to specific user attributes across different domains like emails, abstracts, reviews, and topic-based writing.

## Datasets

- **LongLaMP** — total ?; splits: val (-1), test (-1)

## Metrics

- `ROUGE-1` — range: [0, 1]
  - Unigram overlap between generated and reference text.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated and reference text.
- `METEOR` **(primary)** — range: [0, 1]
  - Precision, recall, and fragmentation penalty based on aligned unigrams.

## Input / output format

**Input**: Task prompt + retrieved user profile snippets (k items) from a retriever (BM25 or Contriever).

**Output**: Generated long-form text corresponding to the task (e.g., email, abstract, review, or topic writing).

## Scoring recipe

```python
import rouge
import nltk
nltk.download('punkt')

def compute_metrics(preds, refs):
    rouge = rouge.Rouge()
    meteor = nltk.translate.meteor_score.single_meteor_score
    scores = {'ROUGE-1': [], 'ROUGE-L': [], 'METEOR': []}
    for p, r in zip(preds, refs):
        r1 = rouge.get_scores(p, r)[0]['rouge-1']['f']
        rL = rouge.get_scores(p, r)[0]['rouge-l']['f']
        m = meteor([r.split()], p.split())
        scores['ROUGE-1'].append(r1)
        scores['ROUGE-L'].append(rL)
        scores['METEOR'].append(m)
    return {k: sum(v)/len(v) for k, v in scores.items()}
```

## Common pitfalls

- Retriever choice (BM25 vs Contriever) and k value significantly impact scores; optimal k varies by domain and setting.
- Temporal vs User settings require different train/test splits based on profile recency or user identity, which must be strictly separated to avoid data leakage.
- GPT-3.5 was not evaluated on the Email domain due to dataset privacy restrictions.

## Evidence (verbatim from paper)

> For zero-shot experiments, GPT-3.5 and LLaMA2 are utilized, and the evaluation metrics used are ROUGE-1, ROUGE-L, and METEOR. To evaluate the results, we assessed the generated output corresponding to each input against the expected output, as described in Section [3].

## Citation

```bibtex
@misc{kumar2024longlamp,
  title={LongLaMP: A Benchmark for Personalized Long-form Text Generation},
  author={Kumar et al. (2024)},
  year={2024},
  note={arXiv:2407.11016}
}
```

- arXiv: 2407.11016