Llmrec Eval

---
name: llmrec-eval
description: Benchmarks off-the-shelf large language models on five recommendation tasks: rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. It probes both accuracy-driven prediction capabilities and natural language generation for explainability, revealing gaps between objective metric scores and human-perceived quality in recommendation contexts. Use when the user wants to benchmark on LLMRec Benchmark (includes Beauty dataset), or asks about evaluating this task. Reports HR@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.12241
  bibtex_key: liu2023llmrec
  confidence: high
---

# llmrec-eval

> LLMRec: Benchmarking Large Language Models on Recommendation Task — Liu et al. (2023) (arXiv:2308.12241, 2023)

## What this evaluates

Benchmarks off-the-shelf large language models on five recommendation tasks: rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. It probes both accuracy-driven prediction capabilities and natural language generation for explainability, revealing gaps between objective metric scores and human-perceived quality in recommendation contexts.

## Datasets

- **LLMRec Benchmark (includes Beauty dataset)** — total ?; splits: test (-1); repo https://github.com/williamliujl/LLMRec

## Metrics

- `RMSE` — range: other
  - Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)).
- `MAE` — range: other
  - Mean Absolute Error: mean(|y_true - y_pred|).
- `HR@5` — range: [0, 1]
  - Hit Rate at K: fraction of test items appearing in the top-K predicted list.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K: measures ranking quality with position-based discounting.
- `HR@10` **(primary)** — range: [0, 1]
  - Hit Rate at K: fraction of test items appearing in the top-K predicted list.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K: measures ranking quality with position-based discounting.
- `ROUGE1` — range: [0, 1]
  - Recall of unigram overlap between generated and reference text.
- `ROUGE2` — range: [0, 1]
  - Recall of bigram overlap between generated and reference text.
- `ROUGEL` — range: [0, 1]
  - Recall of longest common subsequence overlap between generated and reference text.

## Input / output format

**Input**: Textual prompts containing user/item context or interaction history, requesting a numerical rating, a ranked list of items, or a natural language explanation/summary.

**Output**: Numerical value (rating), comma-separated or list of item IDs/names, or free-form natural language text.

## Scoring recipe

```python
def score(preds, golds, K=10):
    rmse = np.sqrt(np.mean((np.array(preds) - np.array(golds))**2))
    mae = np.mean(np.abs(np.array(preds) - np.array(golds)))
    hr = sum(1 for p, g in zip(preds, golds) if g in p[:K]) / len(preds)
    ndcg = compute_ndcg_at_k(preds, golds, k=K)
    rouge = rouge_score(golds, preds, types=['rouge1', 'rouge2', 'rougeL'])
    return {'RMSE': rmse, 'MAE': mae, 'HR@K': hr, 'NDCG@K': ndcg, 'ROUGE': rouge}
```

## Common pitfalls

- Off-the-shelf LLMs lack item-specific exposure, causing poor accuracy in rating and sequential recommendation tasks.
- Models often fail to produce standard structured outputs following prompt requirements without supervised fine-tuning (SFT).
- Objective metrics like ROUGE/BLEU do not correlate well with human-perceived quality of generated explanations or summaries.

## Evidence (verbatim from paper)

> In the context of explainability-based tasks, it is evident that off-the-shelf models are progressively closing the gap with trained models and, in some cases, superseding base-line algorithms based on certain metrics, as clearly demonstrated in Table 1 and Table 2. For instance, in review summarization task, ChatGPT achieves 4.7177, 0.6924, and 4.2557 on ROUGE1, ROUGE2, and ROUGEL metrics, respectively, exceeding the performance of T0 and GPT-2 trained on Beauty dataset.

## Citation

```bibtex
@misc{liu2023llmrec,
  title={LLMRec: Benchmarking Large Language Models on Recommendation Task},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2308.12241}
}
```

- arXiv: 2308.12241