Regen Eval

---
name: regen-eval
description: Evaluates conversational recommender systems on next-item prediction and joint narrative generation, specifically testing how well models incorporate user interaction history and explicit natural language critiques to produce accurate recommendations and contextually grounded textual explanations. Use when the user wants to benchmark on REGEN, or asks about evaluating this task. Reports Recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.11924
  bibtex_key: su2025regen
  confidence: high
---

# regen-eval

> REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives — Su et al. (2025) (arXiv:2503.11924, 2025)

## What this evaluates

Evaluates conversational recommender systems on next-item prediction and joint narrative generation, specifically testing how well models incorporate user interaction history and explicit natural language critiques to produce accurate recommendations and contextually grounded textual explanations.

## Datasets

- **REGEN** — total ?; splits: Office Products (-1), Clothing (-1)

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Fraction of ground-truth next items correctly predicted within the top-10 recommendations.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, averaging the inverse rank of the first correct item across queries.
- `BLEU` — range: percent
  - Bilingual Evaluation Understudy score, measuring n-gram precision between generated and reference narratives.
- `ROUGE` — range: percent
  - Recall-Oriented Understudy for Gisting Evaluation, measuring n-gram recall/overlap between generated and reference narratives.
- `Sem. Sim.` — range: [0, 1]
  - Semantic Similarity, typically cosine similarity between sentence embeddings of generated and reference narratives.

## Input / output format

**Input**: User interaction history sequence $S$ (item IDs, titles, descriptions, categories, and user-generated text) optionally augmented with explicit user critique commands $C$.

**Output**: A single item ID token (`ID_TOKEN`) followed by a natural language narrative $\mathcal{T}_{n+1}$.

## Scoring recipe

```python
def score(predictions, gold):
    rec_preds = [p[0] for p in predictions]
    rec_golds = [g[0] for g in gold]
    pred_texts = [p[1] for p in predictions]
    gold_texts = [g[1] for g in gold]
    recall = compute_recall_at_k(rec_preds, rec_golds, k=10)
    ndcg = compute_ndcg_at_k(rec_preds, rec_golds, k=10)
    mrr = compute_mrr(rec_preds, rec_golds)
    bleu = compute_bleu(pred_texts, gold_texts)
    rouge = compute_rouge(pred_texts, gold_texts)
    sem_sim = compute_semantic_similarity(pred_texts, gold_texts)
    return {'Recall@10': recall, 'NDCG@10': ndcg, 'MRR': mrr, 'BLEU': bleu, 'ROUGE': rouge, 'Sem. Sim.': sem_sim}
```

## Common pitfalls

- Models are evaluated on two distinct architectures (hybrid vs. pure LLM) with different training regimes (separate vs. end-to-end), making direct comparison sensitive to training setup.
- Narrative generation is evaluated with an upper-bound oracle where the true next item ID is provided to the Flare+LLM baseline, which may inflate narrative quality compared to the autoregressive LUMEN model.
- Semantic similarity is reported without specifying the embedding model or distance metric used, making cross-study comparison difficult.

## Evidence (verbatim from paper)

> Table 4. Office Products Dataset - Benchmarks with Joint Recommendation and Generation Tasks. To evaluate upper bound performance, we provide the true next item $\mathrm{ID}_{t+1}$ when generating Flare+LLM narratives.

| ConversationalTask |  | Purchase | | User | | Product | | Purchase | | Long User | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| | | Reason | | Summary | | Endorsement | | Reason Expl. | | Summary | |
| Metrics | Flare+LLM | LUMEN | Flare+LLM | LUMEN | Flare+LLM | LUMEN | Flare+LLM | LUMEN | Flare+LLM | LUMEN |
| Hist.$\downarrow$ID + Narrative | Recall@10 | 0.124 | 0.098 | 0.124 | 0.10 | 0.124 | 0.10 | 0.124 | 0.091 | 0.124 | 0.098 |

## Citation

```bibtex
@misc{su2025regen,
  title={REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2503.11924}
}
```

- arXiv: 2503.11924