Rlirank Eval

---
name: rlirank-eval
description: Evaluates a reinforcement learning framework for dynamic search ranking that adapts to evolving user intents over multiple search iterations using sequential feedback. It also benchmarks standard learning-to-rank performance on static datasets. Use when the user wants to benchmark on TREC 2016 Dynamic Domain, TREC 2017 Dynamic Domain, MQ2007, MQ2008, or asks about evaluating this task. Reports α-NDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.10124
  bibtex_key: zhou2021rlirank
  confidence: high
---

# rlirank-eval

> RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search — Zhou et al. (2021) (arXiv:2105.10124, 2021)

## What this evaluates

Evaluates a reinforcement learning framework for dynamic search ranking that adapts to evolving user intents over multiple search iterations using sequential feedback. It also benchmarks standard learning-to-rank performance on static datasets.

## Datasets

- **TREC 2016 Dynamic Domain** — total ?; splits: test (-1)
- **TREC 2017 Dynamic Domain** — total ?; splits: test (-1)
- **MQ2007** — total ?; splits: test (-1)
- **MQ2008** — total ?; splits: test (-1)

## Metrics

- `α-NDCG` **(primary)** — range: [0, 1]
  - Alpha-NDCG, a variant of NDCG that accounts for position bias and user stopping behavior in dynamic search.
- `nSDCG` — range: [0, 1]
  - Normalized Sequential Discounted Cumulative Gain, measuring ranking quality over sequential search iterations.
- `NDCG@k` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff k (k=1,2,3,4,5), standard metric for static learning-to-rank.

## Input / output format

**Input**: Query, document features, and sequential history of retrieved documents and user feedback across search iterations.

**Output**: Ranked list of documents for each query.

## Scoring recipe

```python
def score(predictions, gold, metric, k=None):
    if metric == 'NDCG@k':
        return ndcg_at_k(predictions, gold, k)
    elif metric == 'α-NDCG':
        return alpha_ndcg(predictions, gold)
    elif metric == 'nSDCG':
        return nscg(predictions, gold)
    return mean([score(q_preds, q_gold, metric) for q_preds, q_gold in zip(all_preds, all_gold)])
```

## Common pitfalls

- Confusing α-NDCG with standard NDCG; α-NDCG incorporates position bias and stopping probability.
- Failing to report per-iteration results for dynamic search datasets, which are crucial for evaluating adaptation.
- Using different NDCG cutoffs (k=1 to 5) without specifying which one is the primary metric.

## Evidence (verbatim from paper)

> Table 1: α-NDCG of baselines, and RLIRank on 2016 TREC Dynamic Domain dataset and nSDCG of ictnet-params2-ns, ictne-emulti, galago-baseline, dqn-5-actions, clip-baseline, and RLIRank on 2017 TREC Dynamic Domain dataset. ... Table 3: NDCG of RankSVM, ListNet, AdaRank-NDCG, MDP, and RLIRank on MQ2007 and MQ2008 dataset.

## Citation

```bibtex
@misc{zhou2021rlirank,
  title={RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search},
  author={Zhou et al. (2021)},
  year={2021},
  note={arXiv:2105.10124}
}
```

- arXiv: 2105.10124