Trec2022 Fair Ranking Eval

---
name: trec2022-fair-ranking-eval
description: Evaluates retrieval systems on balancing topical relevance with intersectional fairness in Wikipedia article rankings. It probes static single-query ranking for coordinators and dynamic multi-query ranking for editors under fairness constraints across demographic attributes. Use when the user wants to benchmark on TREC 2022 Fair Ranking Track, or asks about evaluating this task. Reports M1, EE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.05558
  bibtex_key: ekstrand2023trec2022fairranking
  confidence: high
---

# trec2022-fair-ranking-eval

> Overview of the TREC 2022 Fair Ranking Track — Ekstrand et al. (2023) (arXiv:2302.05558, 2023)

## What this evaluates

Evaluates retrieval systems on balancing topical relevance with intersectional fairness in Wikipedia article rankings. It probes static single-query ranking for coordinators and dynamic multi-query ranking for editors under fairness constraints across demographic attributes.

## Datasets

- **TREC 2022 Fair Ranking Track** — total ?; splits: test (-1); repo https://github.com/fair-trec/trec2022-fair-public

## Metrics

- `nDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at a fixed cutoff, measuring ranking quality based on graded relevance labels.
- `AWRF` — range: [0, 1]
  - Average Weighted Rank Fairness, quantifying the disparity in expected exposure across protected demographic attributes.
- `M1` **(primary)** — range: [0, 1]
  - Official Task 1 composite metric that balances nDCG and AWRF to evaluate the relevance-fairness tradeoff for static rankings.
- `EE-L` **(primary)** — range: other
  - Expected Exposure - Long term, measures long-term document exposure across a sequence of rankings. Lower values indicate better fairness.
- `EE-D` — range: other
  - Expected Exposure - Disparity, measures the variance or inequality in exposure across protected groups.
- `EE-R` — range: [0, 1]
  - Expected Exposure - Relevance, measures the relevance-weighted exposure of documents.

## Input / output format

**Input**: Query string and a collection of Wikipedia articles. Task 1 provides 500 articles per query; Task 2 provides a pool of candidate rankings for 100 queries.

**Output**: Task 1: A single ranked list of 500 documents. Task 2: A sequence of 100 ranked lists, each containing 20 documents.

## Scoring recipe

```python
def score_task1(predictions, gold, protected_attrs):
    ndcg = compute_ndcg(predictions, gold)
    awrf = compute_awrf(predictions, protected_attrs)
    m1 = composite(ndcg, awrf)  # Official formula combines both
    return {'nDCG': ndcg, 'AWRF': awrf, 'M1': m1}

def score_task2(predictions_seq, gold, protected_attrs):
    ee_r = compute_expected_exposure_relevance(predictions_seq, gold)
    ee_d = compute_expected_exposure_disparity(predictions_seq, protected_attrs)
    ee_l = compute_expected_exposure_longterm(predictions_seq)
    return {'EE-R': ee_r, 'EE-D': ee_d, 'EE-L': ee_l}
```

## Common pitfalls

- EE-L and EE-D are minimized (lower is better), whereas nDCG and EE-R are maximized.
- Fairness must be evaluated intersectionally across multiple demographic attributes (age, gender, etc.), not just single attributes.
- Task 1 evaluates static rankings per query, while Task 2 evaluates dynamic exposure across a sequence of 100 queries.

## Evidence (verbatim from paper)

> Table 1 shows the submitted systems ranked by the official Task 1 metric M1 and its component parts nDCG and AWRF. ... Table 4 shows the submitted systems ranked by the official Task 2 metric EE-L and its component parts EE-D and EE-R.

## Citation

```bibtex
@misc{ekstrand2023trec2022fairranking,
  title={Overview of the TREC 2022 Fair Ranking Track},
  author={Ekstrand et al. (2023)},
  year={2023},
  note={arXiv:2302.05558}
}
```

- arXiv: 2302.05558