Trec Session Tracks Eval

---
name: trec-session-tracks-eval
description: Evaluates the effectiveness of a learning-to-rank personalization approach for web search sessions by measuring relevance prediction quality across multiple years of session track data. It probes the model's ability to leverage historical query sequences, document rankings, and user click behavior to improve session-level relevance ranking. Use when the user wants to benchmark on TREC 2011-2014 Session Tracks, or asks about evaluating this task. Reports nDCG@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.08206
  bibtex_key: [REDACTED]
  confidence: high
---

# trec-session-tracks-eval

> Learning to Personalize for Web Search Sessions — Aloteibi et al. (2020) (arXiv:2009.08206, 2020)

## What this evaluates

Evaluates the effectiveness of a learning-to-rank personalization approach for web search sessions by measuring relevance prediction quality across multiple years of session track data. It probes the model's ability to leverage historical query sequences, document rankings, and user click behavior to improve session-level relevance ranking.

## Datasets

- **TREC 2011-2014 Session Tracks** — total 1282; splits: train (-1), val (-1), test (-1)

## Metrics

- `nDCG@k` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank k; computed as DCG@k divided by the ideal DCG@k for the given relevance judgments. DCG sums (2^rel - 1) / log2(rank + 1) over the top-k results.
- `nERR@k` — range: [0, 1]
  - Normalized Expected Reciprocal Rank at rank k; measures the probability that a user stops at a relevant document within the top-k results, normalized by the ideal session.
- `MAP` — range: [0, 1]
  - Mean Average Precision; averages the precision values computed at ranks where relevant documents are retrieved across all queries/sessions.

## Input / output format

**Input**: Session context including the sequence of past queries, ranked document lists for past queries, user clicking behavior, and the current test query.

**Output**: Ranked list of candidate documents (truncated at rank 100) with predicted relevance scores.

## Scoring recipe

```python
import math
def compute_ndcg_at_k(pred_scores, rel_labels, k):
    ranked_indices = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)
    dcg = sum((2**rel_labels[i] - 1) / math.log2(rank + 2) for rank, i in enumerate(ranked_indices[:k]))
    ideal_labels = sorted(rel_labels, reverse=True)
    idcg = sum((2**ideal_labels[i] - 1) / math.log2(i + 2) for i in range(min(k, len(ideal_labels))))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Using standard single-query evaluation instead of session-aware metrics like nERR@k that account for whole-session relevance judgments.
- Failing to use the official TREC evaluation script, which implements specific discounting functions and session-level aggregation conventions.
- Splitting data by documents rather than queries for cross-validation, which causes data leakage in session search benchmarks.

## Evidence (verbatim from paper)

> The evaluation metrics used in this paper are based on TREC session track's official metrics. These are: nDCG@k, nERR@k and MAP. All runs are evaluated using the official evaluation script.

## Citation

```bibtex
@misc{aloteibi2020learning,
  title={Learning to Personalize for Web Search Sessions},
  author={Aloteibi et al. (2020)},
  year={2020},
  note={arXiv:2009.08206}
}
```

- arXiv: 2009.08206