Trec Data Fusion Eval

---
name: trec-data-fusion-eval
description: Evaluates the effectiveness of linear combination data fusion methods for information retrieval when trained on partial relevance judgments. It probes whether multiple linear regression can learn near-optimal fusion weights using only 20% or 50% of relevant documents instead of full official qrels. Use when the user wants to benchmark on TREC 2018-2021 Precision Medicine & Deep Learning Tracks, or asks about evaluating this task. Reports MAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.04981
  bibtex_key: [REDACTED]
  confidence: high
---

# trec-data-fusion-eval

> Streamlined Data Fusion: Unleashing the Power of Linear Combination with Minimal Relevance Judgments — Xu et al. (2023) (arXiv:2309.04981, 2023)

## What this evaluates

Evaluates the effectiveness of linear combination data fusion methods for information retrieval when trained on partial relevance judgments. It probes whether multiple linear regression can learn near-optimal fusion weights using only 20% or 50% of relevant documents instead of full official qrels.

## Datasets

- **TREC 2018-2021 Precision Medicine & Deep Learning Tracks** — total ?; splits: (unstated)

## Metrics

- `MAP` **(primary)** — range: [0, 1]
  - Mean Average Precision across all queries. Averages the average precision (AP) for each query, where AP is the mean of precision values at ranks where relevant documents appear.
- `RP` — range: [0, 1]
  - Reciprocal Precision (system-oriented metric; exact formula not detailed beyond acronym in the text).
- `P@10` — range: [0, 1]
  - Precision at rank 10. The fraction of relevant documents in the top 10 positions of the fused ranked list.
- `P@20` — range: [0, 1]
  - Precision at rank 20. The fraction of relevant documents in the top 20 positions of the fused ranked list.

## Input / output format

**Input**: Per query: ranked document lists from multiple retrieval systems (19, 14, 15, or 16 runs per dataset). Relevance labels (official or partial qrels) are provided for training fusion weights.

**Output**: Per query: a fused ranked list of documents generated by linear combination of reciprocal rank scores from the input runs.

## Scoring recipe

```python
def compute_ir_metrics(pred_ranks, gold_rels):
    ap_sum, p10_sum, p20_sum = 0.0, 0.0, 0.0
    rel_count = 0
    for i, doc in enumerate(pred_ranks):
        if gold_rels.get(doc, 0) == 1:
            rel_count += 1
            ap_sum += rel_count / (i + 1)
        if i == 9:
            p10_sum = rel_count / 10
        if i == 19:
            p20_sum = rel_count / 20
    n_queries = len(pred_ranks)
    return {
        'MAP': ap_sum / n_queries,
        'P@10': p10_sum / n_queries,
        'P@20': p20_sum / n_queries
    }
```

## Common pitfalls

- Using raw retrieval scores instead of the specified reciprocal rank score $score(d)=1/(60+rank(d))$ for fusion.
- Training fusion weights on full official qrels instead of the specified 20% or 50% partial pools selected via fixed-length pooling.
- Failing to split queries into odd/even partitions for the two-fold cross-validation setup.

## Evidence (verbatim from paper)

> Four metrics were used for evaluation: MAP, RP, P@10, and P@20. MAP and RP are system-oriented metrics, while P@10 and P@20 are user-oriented metrics. The two-fold cross-validation methodology was applied: for all the queries in a data set, we divided them into two partitions: odd-numbered and even-numbered. One partition was used for weights training and the other for testing, and vice versa.

## Citation

```bibtex
@misc{xu2023streamlined,
  title={Streamlined Data Fusion: Unleashing the Power of Linear Combination with Minimal Relevance Judgments},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2309.04981}
}
```

- arXiv: 2309.04981