Dl21 Dl22 Ir Eval

---
name: dl21-dl22-ir-eval
description: This protocol evaluates information retrieval systems by measuring their ranking effectiveness on passage retrieval tasks using both original seed queries and LLM-generated query variants aligned with specific demographic or textual profiles. It probes whether retrieval systems perform consistently across diverse user personas and query transformations, revealing potential disparities in system behavior and ranking stability. Use when the user wants to benchmark on DL21 & DL22 (TREC Deep Learning Track), or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.17644
  bibtex_key: alaofi2025demographically
  confidence: high
---

# dl21-dl22-ir-eval

> Demographically-Inspired Query Variants Using an LLM — Alaofi et al. (2025) (arXiv:2508.17644, 2025)

## What this evaluates

This protocol evaluates information retrieval systems by measuring their ranking effectiveness on passage retrieval tasks using both original seed queries and LLM-generated query variants aligned with specific demographic or textual profiles. It probes whether retrieval systems perform consistently across diverse user personas and query transformations, revealing potential disparities in system behavior and ranking stability.

## Datasets

- **DL21 & DL22 (TREC Deep Learning Track)** — total 129; splits: test (129); repo https://github.com/MarwahAlaofi/demo-qv

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 10. Computed over graded relevance judgments on a 4-point scale (0–3). The paper notes that for some comparative analyses, scores are binarized (0,1→0; 2,3→1), but NDCG@10 itself uses the graded scale. Formula: NDCG@10 = DCG@10 / IDCG@10, where DCG@10 = sum_{i=1}^{10} (2^{rel_i} - 1) / log2(i + 1).
- `Cohen's kappa` — range: [-1, 1]
  - Inter-annotator agreement metric used to validate LLM-generated relevance labels against human NIST judgments on a binary scale. Accounts for chance agreement.
- `Krippendorf's alpha` — range: [-1, 1]
  - Reliability metric used to validate LLM-generated relevance labels against human NIST judgments on a 4-point ordinal scale. Measures agreement across multiple annotators/labels while correcting for chance.

## Input / output format

**Input**: Query variants (generated from seed queries + transformation profiles) paired with a passage corpus (MS MARCO v2). For relevance labeling, backstories representing information needs are paired with retrieved passages.

**Output**: Top-10 ranked passages per query variant. Graded relevance labels on a 4-point scale (0–3) assigned by GPT-4o or human annotators.

## Scoring recipe

```python
def ndcg_at_10(gold_relevance, k=10):
    import math
    dcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(gold_relevance[:k]))
    ideal = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0
# Note: For binarized analyses, map rel in {0,1} -> 0, {2,3} -> 1 before scoring.
```

## Common pitfalls

- High rate of missing relevance judgments in top-10 results (41-48%), requiring careful handling or LLM-based relabeling to avoid biased NDCG estimates.
- LLM relevance labeling can exhibit bias favoring seed queries over variants unless mitigated by generating contextual backstories instead of using raw seed queries.
- Query variant generation temperature must be set to 1.0; lower temperatures produce deterministic but repetitive outputs that fail to capture lexical diversity.

## Evidence (verbatim from paper)

> Evaluation is based on NDCG@10, the official metric for DL21 (Deep Learning Track of TREC 2021) and DL22 (Deep Learning Track of TREC 2022). Systems were evaluated across all variant sets: seed, persona-based, user group-based, and textual transformation sets. As anticipated, we encountered a substantial portion of missing relevance judgments in the top ten results retrieved in response to query variant sets. On average, at a cutoff of 10, 41% and 48% of relevance judgments are missing in DL21 (Deep Learning Track of TREC 2021) and DL22 (Deep Learning Track of TREC 2022), respectively, raising concerns about the evaluation outcomes.

## Citation

```bibtex
@misc{alaofi2025demographically,
  title={Demographically-Inspired Query Variants Using an LLM},
  author={Alaofi et al. (2025)},
  year={2025},
  note={arXiv:2508.17644}
}
```

- arXiv: 2508.17644