Perrecbench Eval

---
name: perrecbench-eval
description: Evaluates whether LLMs can capture true personalized user preferences by ranking items or users in groups, while explicitly controlling for confounding factors like user rating bias and item quality. It probes the model's ability to perform comparative reasoning rather than simple rating prediction. Use when the user wants to benchmark on PerRecBench, or asks about evaluating this task. Reports Kendall’s tau.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.13391
  bibtex_key: tan2025perrecbench
  confidence: high
---

# perrecbench-eval

> Can Large Language Models Understand Preferences in Personalized Recommendation? — Tan et al. (2025) (arXiv:2501.13391, 2025)

## What this evaluates

Evaluates whether LLMs can capture true personalized user preferences by ranking items or users in groups, while explicitly controlling for confounding factors like user rating bias and item quality. It probes the model's ability to perform comparative reasoning rather than simple rating prediction.

## Datasets

- **PerRecBench** — total ?; splits: test (-1); repo https://github.com/TamSiuhin/PerRecBench

## Metrics

- `Kendall’s tau` **(primary)** — range: [-1, 1]
  - Measures the rank correlation between the predicted ordering and the ground truth ordering. Values range from -1 (perfect inverse correlation) to 1 (perfect correlation), with 0 indicating no association.

## Input / output format

**Input**: Grouped prompts containing user and item context, formatted for pointwise (single user), pairwise (two users), or listwise (multiple users) ranking tasks.

**Output**: A ranked list of users or items according to the model's predicted preference order.

## Scoring recipe

```python
def kendall_tau(pred, true):
    n = len(pred)
    concordant = discordant = 0
    for i in range(n):
        for j in range(i + 1, n):
            diff_pred = pred[i] - pred[j]
            diff_true = true[i] - true[j]
            if diff_pred * diff_true > 0:
                concordant += 1
            elif diff_pred * diff_true < 0:
                discordant += 1
    return (concordant - discordant) / (n * (n - 1) / 2)
```

## Common pitfalls

- Pointwise ranking methods evaluate users in isolation, making it difficult for models to capture subtle preference differences compared to pairwise or listwise approaches.
- Larger model size does not consistently improve personalization performance, challenging standard scaling law assumptions.
- Traditional regression metrics like rating prediction show low correlation with actual personalized ranking ability, so high prediction scores do not guarantee good personalization.

## Evidence (verbatim from paper)

> Across 19 strong LLMs, performance on PerRecBench ranges from 0.02 to 0.18, within Kendall’s tau value range of $[-1,1]$. This indicates a low to moderate correlation between predictions and ground truth rankings.

## Citation

```bibtex
@misc{tan2025perrecbench,
  title={Can Large Language Models Understand Preferences in Personalized Recommendation?},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2501.13391}
}
```

- arXiv: 2501.13391