Ncf Implicit Rec Eval

---
name: ncf-implicit-rec-eval
description: Evaluates a model's ability to predict implicit user-item interactions and rank relevant items for recommendation. It probes non-linear collaborative filtering capabilities on sparse, implicit feedback datasets by measuring whether the true interacted item appears near the top of a ranked list. Use when the user wants to benchmark on MovieLens, Pinterest, or asks about evaluating this task. Reports HR@10, NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1708.05031
  bibtex_key: he2017neural
  confidence: high
---

# ncf-implicit-rec-eval

> Neural Collaborative Filtering — Xiangnan He et al. (arXiv:1708.05031, 2017)

## What this evaluates

Evaluates a model's ability to predict implicit user-item interactions and rank relevant items for recommendation. It probes non-linear collaborative filtering capabilities on sparse, implicit feedback datasets by measuring whether the true interacted item appears near the top of a ranked list.

## Datasets

- **MovieLens** — total 1000209; splits: train (-1), test (6040)
- **Pinterest** — total 1500809; splits: train (-1), test (55187)

## Metrics

- `HR@10` **(primary)** — range: [0, 1]
  - Hit Ratio at rank 10. Returns 1.0 if the ground-truth test item is ranked within the top 10 positions of the candidate list, and 0.0 otherwise. Averaged across all test users.
- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes 1/log2(rank+1) if the test item is in the top 10, else 0. Normalized by the ideal DCG (which is 1 when the test item is ranked first). Averaged across all test users.

## Input / output format

**Input**: User ID and Item ID pairs representing implicit interactions (binary 0/1). During evaluation, a user's historical interactions are used to generate a ranked list of candidate items.

**Output**: A ranked list of items (or predicted relevance scores) for each user, evaluated against a held-out test item.

## Scoring recipe

```python
def evaluate(user_history, test_item, sampled_negatives, k=10):
    candidates = sampled_negatives + [test_item]
    scores = model.predict(user_history, candidates)
    ranked = sorted(candidates, key=lambda x: scores[x], reverse=True)
    test_rank = ranked.index(test_item) + 1
    hr = 1.0 if test_rank <= k else 0.0
    ndcg = (1.0 / math.log2(test_rank + 1)) if test_rank <= k else 0.0
    return hr, ndcg
# Average HR and NDCG over all users
```

## Common pitfalls

- The evaluation ranks the test item against only 100 randomly sampled negative items, not the full item catalog. This speeds up computation but means metrics reflect relative ranking within a small candidate set rather than absolute catalog ranking.
- Datasets must be preprocessed to implicit feedback (binary 0/1) and filtered to retain only users with at least 20 interactions before applying the leave-one-out split.
- Training uses a negative sampling ratio of 4 negatives per positive instance, which is distinct from the 100-item evaluation sampling strategy and should not be confused.

## Evidence (verbatim from paper)

> To evaluate the performance of item recommendation, we adopted the leave-one-out evaluation, which has been widely used in literature [1, 14, 27]. For each user, we held-out her latest interaction as the test set and utilized the remaining data for training. Since it is too time-consuming to rank all items for every user during evaluation, we followed the common strategy [6, 21] that randomly samples 100 items that are not interacted by the user, ranking the test item among the 100 items. The performance of a ranked list is judged by Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) [11]. Without special mention, we truncated the ranked list at 10 for both metrics.

## Citation

```bibtex
@misc{he2017neural,
  title={Neural Collaborative Filtering},
  author={Xiangnan He et al.},
  year={2017},
  note={arXiv:1708.05031}
}
```

- arXiv: 1708.05031