Orbit Eval

---
name: orbit-eval
description: Evaluates recommendation models on candidate item ranking across multiple public sequential recommendation datasets and a large-scale synthetic hidden test (ClueWeb-Reco) to assess generalization to unseen item pools and real-world browsing scenarios. Use when the user wants to benchmark on ML-1M, Amazon Beauty, Amazon Toys, Amazon Sports, Amazon Books, ClueWeb-Reco, or asks about evaluating this task. Reports Recall@10, NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.26095
  bibtex_key: he2025orbit
  confidence: high
---

# orbit-eval

> ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests — He et al. (2025) (arXiv:2510.26095, 2025)

## What this evaluates

Evaluates recommendation models on candidate item ranking across multiple public sequential recommendation datasets and a large-scale synthetic hidden test (ClueWeb-Reco) to assess generalization to unseen item pools and real-world browsing scenarios.

## Datasets

- **ML-1M** — total ?; splits: train (-1), test (-1)
- **Amazon Beauty** — total ?; splits: train (-1), test (-1)
- **Amazon Toys** — total ?; splits: train (-1), test (-1)
- **Amazon Sports** — total ?; splits: train (-1), test (-1)
- **Amazon Books** — total ?; splits: train (-1), test (-1)
- **ClueWeb-Reco** — total ?; splits: test (-1)

## Metrics

- `Recall@10` **(primary)** — range: [0, 1]
  - Fraction of relevant items correctly retrieved in the top-10 recommendations. Recall@K = |Predicted ∩ Relevant| / |Relevant|.
- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. NDCG@K = DCG@K / IDCG@K, where DCG@K = Σ_{i=1}^K (rel_i / log2(i+1)).
- `Recall@50` — range: [0, 1]
  - Fraction of relevant items correctly retrieved in the top-50 recommendations.
- `NDCG@50` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 50.
- `Recall@100` — range: [0, 1]
  - Fraction of relevant items correctly retrieved in the top-100 recommendations.
- `NDCG@100` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 100.

## Input / output format

**Input**: User interaction history (sequential clicks/purchases) and a candidate item pool.

**Output**: A ranked list of recommended items (top-K).

## Scoring recipe

```python
def compute_metrics(pred_list, relevant_set, k):
    top_k = pred_list[:k]
    recall = len(set(top_k) & relevant_set) / len(relevant_set)
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(top_k) if item in relevant_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(k, len(relevant_set))))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return recall, ndcg
```

## Common pitfalls

- The ClueWeb-Reco hidden test evaluates zero-shot generalization to a massive, unseen item pool, making direct comparison with models trained on the same distribution misleading.
- Metrics are reported at multiple cutoffs (10, 50, 100); focusing solely on NDCG@10 may overlook deep-list recall capabilities crucial for real-world recommendation.
- Content-based and LLM baselines leverage rich item metadata, whereas ID-based models rely purely on interaction sequences; evaluation must account for this architectural difference.

## Evidence (verbatim from paper)

> Table 4 presents Recall@10 and NDCG@10 results, with key observations below: (1) We witness a consistent performance gain in sequential-based ID-based models through their evolution from RNN-based architecture to transformer architecture due to the attention-based structure is better at discovering long-term behavior patterns and user interests.

## Citation

```bibtex
@misc{he2025orbit,
  title={ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2510.26095}
}
```

- arXiv: 2510.26095