Ma4div Diversity Eval

---
name: ma4div-diversity-eval
description: Evaluates a model's ability to rerank search results to maximize topic diversity, ensuring that top-k results cover multiple relevant subtopics for a given query rather than just maximizing single-topic relevance. Use when the user wants to benchmark on TREC 2009~2012 Web Track, DU-DIV, or asks about evaluating this task. Reports α-NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17421
  bibtex_key: chen2024ma4div
  confidence: high
---

# ma4div-diversity-eval

> MA4DIV: Multi-Agent Reinforcement Learning for Search Result Diversification — Chen et al. (2024) (arXiv:2403.17421, 2024)

## What this evaluates

Evaluates a model's ability to rerank search results to maximize topic diversity, ensuring that top-k results cover multiple relevant subtopics for a given query rather than just maximizing single-topic relevance.

## Datasets

- **TREC 2009~2012 Web Track** — total 198; splits: train (-1), test (-1)
- **DU-DIV** — total 4473; splits: train (3578), test (895)

## Metrics

- `α-NDCG@10` **(primary)** — range: [0, 1]
  - Discounted Cumulative Gain that penalizes redundancy by weighting relevance by the fraction of new subtopics covered at each position, normalized by ideal DCG.
- `ERR-IA@10` — range: [0, 1]
  - Expected Reciprocal Rank for Information Adequacy, computing the probability that a user finds a relevant subtopic at each rank position.
- `S-recall@10` — range: [0, 1]
  - Subtopic Recall, measuring the fraction of all relevant subtopics for a query that appear in the top-k ranked documents.
- `α-NDCG@5` — range: [0, 1]
  - Same as α-NDCG@10 but evaluated at cut-off k=5.
- `ERR-IA@5` — range: [0, 1]
  - Same as ERR-IA@10 but evaluated at cut-off k=5.
- `S-recall@5` — range: [0, 1]
  - Same as S-recall@10 but evaluated at cut-off k=5.

## Input / output format

**Input**: A search query and a candidate list of documents (typically top-15 or subsampled 30) with pre-computed embeddings.

**Output**: A ranked permutation of the candidate documents.

## Scoring recipe

```python
def compute_alpha_ndcg(ranked_docs, doc_subtopics, k):
    covered = set()
    dcg = 0.0
    for i, doc in enumerate(ranked_docs[:k]):
        rel = len(doc_subtopics[doc])
        new_topics = len(doc_subtopics[doc] - covered)
        covered.update(doc_subtopics[doc])
        dcg += (2**rel - 1) / math.log2(i + 2) * (new_topics / rel if rel > 0 else 0)
    # idcg computed similarly on ideal ranking
    return dcg / idcg
```

## Common pitfalls

- The TREC dataset is extremely small (198 queries), causing severe overfitting; the authors subsample to 30 documents per query and use 5-fold cross-validation to mitigate this.
- Embedding models differ across datasets (Doc2vec for TREC vs. BERT for DU-DIV), making cross-dataset metric comparisons potentially confounded by representation quality.
- Statistical significance is only reported for comparisons against MA4DIV (marked with '*'), not for all baseline pairs.

## Evidence (verbatim from paper)

> Table 3 and 4 report the performances of all baselines and our MA4DIV in terms of the six diversity performance metrics, including α-NDCG@5, α-NDCG@10, ERR-IA@5, ERR-IA@10, S-recall@5, S-recall@10 on the TREC web track datasets and the DU-DIV dataset respectively.

## Citation

```bibtex
@misc{chen2024ma4div,
  title={MA4DIV: Multi-Agent Reinforcement Learning for Search Result Diversification},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.17421}
}
```

- arXiv: 2403.17421