Fairco Dynamic Ltr Eval

---
name: fairco-dynamic-ltr-eval
description: This evaluation probes a dynamic learning-to-rank algorithm's ability to balance ranking quality with group-level fairness under position bias. It tests whether the model can maintain high relevance-based ranking performance while actively controlling exposure and impact disparities between predefined item groups over a sequence of user interactions. Use when the user wants to benchmark on Ad Fontes Media Bias (semi-synthetic news), MovieLens-20M, or asks about evaluating this task. Reports average cumulative NDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.14713
  bibtex_key: [REDACTED]
  confidence: high
---

# fairco-dynamic-ltr-eval

> Controlling Fairness and Bias in Dynamic Learning-to-Rank — Morik et al. (2020) (arXiv:2005.14713, 2020)

## What this evaluates

This evaluation probes a dynamic learning-to-rank algorithm's ability to balance ranking quality with group-level fairness under position bias. It tests whether the model can maintain high relevance-based ranking performance while actively controlling exposure and impact disparities between predefined item groups over a sequence of user interactions.

## Datasets

- **Ad Fontes Media Bias (semi-synthetic news)** — total ?; splits: simulation (-1); repo https://github.com/MarcoMorik/Dynamic-Fairness
- **MovieLens-20M** — total ?; splits: simulation (-1)

## Metrics

- `average cumulative NDCG` **(primary)** — range: [0, 1]
  - Average over time $\tau$ of the DCG utility $U^{DCG}(\sigma_t|\mathbf{r}_t)$ computed on the true relevance vector $\mathbf{r}_t$ and ranking $\sigma_t$. Normalized by ideal DCG.
- `Exposure Unfairness` — range: other
  - Group disparity metric $\overline{D}_{\tau}^{E}$ measuring differences in cumulative exposure allocation between groups over time $\tau$ (defined in Eq. 16).
- `Impact Unfairness` — range: other
  - Group disparity metric $\overline{D}_{\tau}^{I}$ measuring differences in cumulative impact (relevance-weighted exposure) between groups over time $\tau$ (defined in Eq. 16).

## Input / output format

**Input**: A set of items (news articles or movies) with group labels and features, plus a user context vector (polarity/openness parameters for news, or learned embeddings for movies).

**Output**: A ranking permutation $\sigma_t$ of the items for the current user.

## Scoring recipe

```python
def compute_metrics(rankings, true_relevances, group_labels, tau):
    ndcg_sum = 0.0
    group_exposure = {g: 0.0 for g in set(group_labels)}
    for t in range(tau):
        # NDCG calculation
        dcg = sum(true_relevances[t][i] / log2(i + 2) for i in range(len(rankings[t])))
        idcg = sum(sorted(true_relevances[t], reverse=True)[i] / log2(i + 2) for i in range(len(rankings[t])))
        ndcg_sum += dcg / idcg if idcg > 0 else 0
        # Exposure tracking for unfairness
        for pos, item in enumerate(rankings[t]):
            exposure = 1.0 / log2(pos + 2)
            group_exposure[group_labels[item]] += exposure
    avg_ndcg = ndcg_sum / tau
    # Simplified proxy for Eq. 16 disparity
    unfairness = abs(group_exposure['left'] - group_exposure['right'])
    return avg_ndcg, unfairness
```

## Common pitfalls

- Confusing Exposure Unfairness with Impact Unfairness; the paper shows optimizing one can increase the other (Fig. 10).
- Assuming debiased relevance estimation (IPS) alone ensures fairness; it corrects position bias but does not control group disparity without an explicit fairness controller.
- Treating the semi-synthetic news simulation as a direct substitute for real-world click logs; user behavior is strictly modeled via a Position-based Click Model (PBM) with synthetic polarities.

## Evidence (verbatim from paper)

> We measure ranking quality by the average cumulative NDCG $\frac{1}{\tau}\sum_{t=1}^{\tau}U^{DCG}(\sigma_t|\mathbf{r}_t)$ over all the users up to time $\tau$. We measure Exposure Unfairness via $\overline{D}_{\tau}^{E}$ and Impact Unfairness via $\overline{D}_{\tau}^{I}$ as defined in Equation (16).

## Citation

```bibtex
@misc{morik2020controllingfairness,
  title={Controlling Fairness and Bias in Dynamic Learning-to-Rank},
  author={Morik et al. (2020)},
  year={2020},
  note={arXiv:2005.14713}
}
```

- arXiv: 2005.14713