Lamp Qa Eval

---
name: lamp-qa-eval
description: This benchmark evaluates a model's ability to generate long-form, personalized question-answering responses by aligning outputs with fine-grained, user-specific information needs extracted from community Q&A narratives. It probes aspect-based response quality rather than binary correctness, measuring how well generated answers address individual criteria tailored to a specific user's profile. Use when the user wants to benchmark on LaMP-QA, or asks about evaluating this task. Reports aspect-based evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.00137
  bibtex_key: salemi2025lampqa
  confidence: high
---

# lamp-qa-eval

> LaMP-QA: A Benchmark for Personalized Long-form Question Answering — Salemi et al. (2025) (arXiv:2506.00137, 2025)

## What this evaluates

This benchmark evaluates a model's ability to generate long-form, personalized question-answering responses by aligning outputs with fine-grained, user-specific information needs extracted from community Q&A narratives. It probes aspect-based response quality rather than binary correctness, measuring how well generated answers address individual criteria tailored to a specific user's profile.

## Datasets

- **LaMP-QA** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/LaMP-Benchmark/LaMP-QA

## Metrics

- `aspect-based evaluation` **(primary)** — range: [0, 1]
  - Scores responses based on how well they address individual aspects extracted from the user's stated information needs (narratives). The LLM evaluator assigns a continuous score between 0 and 1 for each aspect, and the final metric is the average across all extracted aspects.

## Input / output format

**Input**: Question, user narrative/profile context, retrieved profile items (k=10), and the generated response.

**Output**: A continuous score between 0 and 1 representing the degree to which the response addresses the user's aspect-based information needs.

## Scoring recipe

```python
def compute_aspect_based_score(question, narrative, response):
    aspects = extract_aspects_from_narrative(narrative) # LLM extracts fine-grained criteria
    scores = []
    for aspect in aspects:
        score = llm_rater_score(response, aspect) # LLM assigns 0-1 alignment score
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Pairwise LLM evaluation suffers from severe position bias, with preference reversal rates up to 78% when response order is swapped.
- Smaller evaluator LLMs (e.g., 0.5B) act as binary classifiers, inflating scores and misaligning with human judgments compared to larger models (32B).
- Using randomly sampled user profiles instead of the actual asker's profile degrades performance below non-personalized baselines, highlighting the necessity of true user-specific context.

## Evidence (verbatim from paper)

> Lastly, we use our proposed evaluation method in Section[3.2], which scores responses based on how well they address the individual aspects extracted from the user’s information needs. The implementation details for all evaluation approaches are provided in Appendix[C].

## Citation

```bibtex
@misc{salemi2025lampqa,
  title={LaMP-QA: A Benchmark for Personalized Long-form Question Answering},
  author={Salemi et al. (2025)},
  year={2025},
  note={arXiv:2506.00137}
}
```

- arXiv: 2506.00137