Regen Quality Eval

---
name: regen-quality-eval
description: Evaluates whether increasing the length of a user's purchase history context improves recommendation quality for LLM-based agents. It probes the saturation point of personalization reasoning and the cost-efficiency trade-off of context length. Use when the user wants to benchmark on REGEN, or asks about evaluating this task. Reports quality scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.20316
  bibtex_key: chauhan2026less
  confidence: high
---

# regen-quality-eval

> Less is More: Benchmarking LLM Based Recommendation Agents — Chauhan et al. (2026) (arXiv:2601.20316, 2026)

## What this evaluates

Evaluates whether increasing the length of a user's purchase history context improves recommendation quality for LLM-based agents. It probes the saturation point of personalization reasoning and the cost-efficiency trade-off of context length.

## Datasets

- **REGEN** — total 50; splits: test (50)

## Metrics

- `quality scores` **(primary)** — range: [0, 1]
  - A recommendation accuracy metric reported on a scale of approximately 0.16 to 0.23. The exact mathematical formulation is not detailed in the results section, but it measures the relevance of the agent's recommendations against ground truth user interactions.

## Input / output format

**Input**: A list of user purchase history items, truncated or padded to specific context lengths (5, 10, 15, 25, or 50 items).

**Output**: A ranked list of recommended items.

## Scoring recipe

```python
def compute_quality(predictions, gold):
    # Calculate relevance metric between predicted items and ground truth
    score = relevance_metric(predictions, gold)
    return score

# Average across 50 users per context length
avg_quality = mean([compute_quality(preds, gold) for preds, gold in zip(predictions, golds)])
```

## Common pitfalls

- Assuming longer context lengths inherently improve recommendation accuracy without empirical verification.
- Ignoring the significant token cost and latency overhead when using maximal context lengths.
- Overlooking the within-subject experimental design, which requires paired statistical tests rather than independent comparisons.

## Evidence (verbatim from paper)

> Table 2 presents our complete results across all models and context lengths. The key observation is that quality scores remain remarkably stable regardless of context length, ranging from 0.16 to 0.23 across all conditions.

## Citation

```bibtex
@misc{chauhan2026less,
  title={Less is More: Benchmarking LLM Based Recommendation Agents},
  author={Chauhan et al. (2026)},
  year={2026},
  note={arXiv:2601.20316}
}
```

- arXiv: 2601.20316