Kuaisim Eval

---
name: kuaisim-eval
description: Evaluates recommender system policies across three temporal granularities: request-level list-wise ranking, whole-session sequential recommendation under reinforcement learning, and cross-session user retention optimization. It measures how well simulated agents balance immediate engagement rewards with long-term user retention and list diversity. Use when the user wants to benchmark on KuaiRand, ML-1m, or asks about evaluating this task. Reports Average L-reward.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.12645
  bibtex_key: zhao2023kuaisim
  confidence: high
---

# kuaisim-eval

> KuaiSim: A Comprehensive Simulator for Recommender Systems — Zhao et al. (2023) (arXiv:2309.12645, 2023)

## What this evaluates

Evaluates recommender system policies across three temporal granularities: request-level list-wise ranking, whole-session sequential recommendation under reinforcement learning, and cross-session user retention optimization. It measures how well simulated agents balance immediate engagement rewards with long-term user retention and list diversity.

## Datasets

- **KuaiRand** — total ?; splits: train (-1), test (-1)
- **ML-1m** — total ?; splits: train (-1), test (-1)

## Metrics

- `Average L-reward` **(primary)** — range: other
  - Average of item-wise immediate rewards across user requests in a mini-batch.
- `Coverage` — range: other
  - Number of distinct items exposed in a mini-batch.
- `ILD (Intra-list diversity)` — range: other
  - Estimates dissimilarity between items in each recommended list based on item embeddings.
- `Whole-session reward` — range: other
  - Average sum of immediate rewards for each session.
- `Return day` — range: other
  - Average time gap (in days) between the last request of one session and the first request of the next.
- `User retention` — range: percent
  - Average ratio of users visiting the system again after a session.

## Input / output format

**Input**: User interaction history or current session context, along with a candidate item pool.

**Output**: A ranked list of recommended items or a sequence of items for the session.

## Scoring recipe

```python
def evaluate(predictions, gold, embeddings, user_history):
    l_rewards = [sum(item_reward(i) for i in pred) for pred in predictions]
    avg_l = sum(l_rewards) / len(l_rewards)
    max_l = max(l_rewards)
    coverage = len(set(i for pred in predictions for i in pred))
    ild = diversity_score(predictions, embeddings)
    session_rewards = [sum(r for r in sess) for sess in sessions]
    whole_sess = sum(session_rewards) / len(session_rewards)
    depth = max(len(sess) for sess in sessions)
    return_day = avg_gap_days(user_history)
    retention = returning_users / total_users
    return avg_l, max_l, coverage, ild, whole_sess, depth, return_day, retention
```

## Common pitfalls

- Metrics are defined at different granularities (request-level vs session-level vs cross-session), making direct comparison across tasks difficult.
- Reward signals are generated by pretrained simulation models rather than real user clicks, which may introduce distributional shift.
- ILD relies on item embeddings that may not align with human-perceived diversity or business objectives.

## Evidence (verbatim from paper)

> List-wise reward (L-reward) is the average of item-wise immediate reward. We use both the average L-reward and the max L-reward across user requests in a mini-batch. Coverage describes the number of distinct items exposed in a mini-batch. Intra-list diversity (ILD) estimates the dissimilarity between items in each recommended list, based on item embedding.

## Citation

```bibtex
@misc{zhao2023kuaisim,
  title={KuaiSim: A Comprehensive Simulator for Recommender Systems},
  author={Zhao et al. (2023)},
  year={2023},
  note={arXiv:2309.12645}
}
```

- arXiv: 2309.12645