Idsr Eval

---
name: idsr-eval
description: Evaluates the accuracy and diversity of end-to-end sequential recommendation models by testing their ability to predict the next item in a user's behavior sequence while maintaining item diversity in the recommendation list. Use when the user wants to benchmark on ML100K, ML1M, or asks about evaluating this task. Reports Recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.10171
  bibtex_key: chen2019improving
  confidence: high
---

# idsr-eval

> Improving End-to-End Sequential Recommendations with Intent-aware Diversification — Chen et al. (2019) (arXiv:1908.10171, 2019)

## What this evaluates

Evaluates the accuracy and diversity of end-to-end sequential recommendation models by testing their ability to predict the next item in a user's behavior sequence while maintaining item diversity in the recommendation list.

## Datasets

- **ML100K** — total 100000; splits: train (-1), val (-1), test (-1)
- **ML1M** — total 1000209; splits: train (-1), val (-1), test (-1)

## Metrics

- `Recall` **(primary)** — range: [0, 1]
  - Fraction of ground-truth test items that appear in the top-K recommended list.
- `MRR` — range: [0, 1]
  - Reciprocal of the rank of the first relevant item in the recommended list.
- `ILD` — range: other
  - Average pairwise dissimilarity between items in the recommendation list, calculated as the Euclidean distance between their genre vectors.

## Input / output format

**Input**: A sequence of the user's 9 most recent movie interactions, sorted by timestamp.

**Output**: A ranked list of recommended movies (top-K).

## Scoring recipe

```python
def compute_metrics(recommended_list, test_item, genre_vectors):
    recall = 1.0 if test_item in recommended_list else 0.0
    mrr = 0.0
    for rank, item in enumerate(recommended_list, 1):
        if item == test_item:
            mrr = 1.0 / rank
            break
    n = len(recommended_list)
    ild = 0.0
    if n > 1:
        dists = [euclidean_dist(genre_vectors[i], genre_vectors[j]) for i in range(n) for j in range(i+1, n)]
        ild = (2.0 / (n * (n - 1))) * sum(dists)
    return recall, mrr, ild
```

## Common pitfalls

- Diversity evaluation requires item attributes (e.g., genres); datasets containing only IDs cannot be used for diversity metrics.
- The MMR baseline uses a fixed trade-off parameter θ=0.5, which may not be optimal across different datasets.
- Input sequences are strictly truncated to the 9 most recent interactions, potentially ignoring long-term user preferences.

## Evidence (verbatim from paper)

> For accuracy evaluation, we use Recall and MRR (Li et al. 2017; Liu et al. 2018); For diversity evaluation, we use Intra-List Distance (ILD) (Zhang and Hurley 2008): ... ILD is defined as: $$ \mathrm {I L D} = \frac {2}{| R _ {L} | (| R _ {L} | - 1)} \sum_ {(i, j) \in R _ {L}} d _ {i j}. \tag {12} $$ We calculate the dissimilarity $d_{ij}$ between two movies based on Euclidean distance between the item genre vectors of movies (Ashkan et al. 2015).

## Citation

```bibtex
@misc{chen2019improving,
  title={Improving End-to-End Sequential Recommendations with Intent-aware Diversification},
  author={Chen et al. (2019)},
  year={2019},
  note={arXiv:1908.10171}
}
```

- arXiv: 1908.10171