Pixelrec Eval

---
name: pixelrec-eval
description: Evaluates the ability of recommender systems to rank items using raw pixel images instead of traditional ID embeddings. It probes cold-start item recommendation, cross-domain transfer learning, and end-to-end vision-based recommendation performance. Use when the user wants to benchmark on PixelRec, or asks about evaluating this task. Reports Recall@N.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.06789
  bibtex_key: cheng2023pixelrec
  confidence: high
---

# pixelrec-eval

> An Image Dataset for Benchmarking Recommender Systems with Raw Pixels — Cheng et al. (arXiv:2309.06789, 2023)

## What this evaluates

Evaluates the ability of recommender systems to rank items using raw pixel images instead of traditional ID embeddings. It probes cold-start item recommendation, cross-domain transfer learning, and end-to-end vision-based recommendation performance.

## Datasets

- **PixelRec** — total 200000000; splits: train (3565656), val (46546), test (40398); repo https://github.com/westlake-repl/PixelRec

## Metrics

- `Recall@N` **(primary)** — range: [0, 1]
  - Fraction of ground-truth items that appear in the top-N predicted items. Calculated as |R ∩ T| / |T| where R is the recommended set and T is the ground truth. N is set to 5 and 10.
- `NDCG@N` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank N. Measures ranking quality by discounting gains logarithmically based on position: sum_{i=1}^{N} (rel_i / log2(i+1)) / ideal_DCG. N is set to 5 and 10.

## Input / output format

**Input**: Ordered sequence of user interaction history (item IDs or raw cover images) truncated or padded to a maximum length of 10, plus a target item to predict.

**Output**: A ranked list or probability score for every item in the full item pool, sorted by predicted relevance.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, k=10):
    top_k = set(predictions[:k])
    recall = len(top_k & ground_truth) / len(ground_truth)
    ndcg = 0.0
    for i, item in enumerate(predictions[:k]):
        if item in ground_truth:
            ndcg += 1.0 / math.log2(i + 2)
    idcg = 1.0 / math.log2(2)
    ndcg /= idcg
    return recall, ndcg
```

## Common pitfalls

- Ranking must be performed over the entire item pool, not a sampled set of 100 negative items, as explicitly noted in the protocol.
- The leave-one-out split assigns the last interaction to test and penultimate to validation, which can artificially inflate performance if user history is not properly masked during training.
- Sequential models require splitting user sequences into fixed-length subsequences of exactly 10 items for training.

## Evidence (verbatim from paper)

> We apply two popular top-N metrics, i.e. Recall@N and NDCG@N (Normalized Discounted Cumulative Gain), to evaluate recommendation performance. N is set to 5 and 10. Note that we rank the predicted item among all items in the pool instead of drawing 100 random items.

## Citation

```bibtex
@misc{cheng2023pixelrec,
  title={An Image Dataset for Benchmarking Recommender Systems with Raw Pixels},
  author={Cheng et al.},
  year={2023},
  note={arXiv:2309.06789}
}
```

- arXiv: 2309.06789