Blair Retrieval Recommendation Eval

---
name: blair-retrieval-recommendation-eval
description: Evaluates a model's ability to align natural language reviews with item metadata for downstream recommendation and search tasks. It probes sequential next-item prediction, conventional keyword-based product retrieval, and complex long-context product search. Use when the user wants to benchmark on Amazon REVIEWS 2023 (Beauty, Games, Baby), ESCI, Amazon-C4, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.03952
  bibtex_key: hou2024blair
  confidence: high
---

# blair-retrieval-recommendation-eval

> Bridging Language and Items for Retrieval and Recommendation — Hou et al. (2024) (arXiv:2403.03952, 2024)

## What this evaluates

Evaluates a model's ability to align natural language reviews with item metadata for downstream recommendation and search tasks. It probes sequential next-item prediction, conventional keyword-based product retrieval, and complex long-context product search.

## Datasets

- **Amazon REVIEWS 2023 (Beauty, Games, Baby)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/hyp1231/AmazonReviews2023
- **ESCI** — total ?; splits: test (27643)
- **Amazon-C4** — total ?; splits: test (21223)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores (binary 1 for relevant, 0 otherwise) discounted by log2(rank+1), normalized by the ideal DCG for the top 10 results.
- `NDCG@100` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 100. Same as NDCG@10 but evaluated over the top 100 retrieved items.

## Input / output format

**Input**: Sequential recommendation: chronological sequence of historical item IDs paired with their metadata (title, features, description). Product search: user query (short keyword phrase or long complex context) and a candidate pool of item metadata.

**Output**: Ranked list of candidate items (top-K) based on computed relevance scores.

## Scoring recipe

```python
def ndcg_at_k(relevant_items, predicted_ranking, k):
    dcg = 0.0
    for i, item in enumerate(predicted_ranking[:k]):
        if item in relevant_items:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_items), k)))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Splits are created by absolute timestamps (8:1:1 ratio), not random or chronological tail-sampling.
- The authors explicitly avoid 5-core filtering to prevent distribution shift toward popular items.
- For product search, candidate pools are constructed by randomly sampling 50 in-domain items per query, not using the full catalog.

## Evidence (verbatim from paper)

> We evaluate the models on the test set using the model that achieves the best ranking performance (NDCG@10) on the validation set. Table 7: Performance comparison of different methods on conventional product search (ESCI) and complex product search (Amazon-C4) tasks. We report the NDCG@100 metric.

## Citation

```bibtex
@misc{hou2024blair,
  title={Bridging Language and Items for Retrieval and Recommendation},
  author={Hou et al. (2024)},
  year={2024},
  note={arXiv:2403.03952}
}
```

- arXiv: 2403.03952