Trec Covid Eval

---
name: trec-covid-eval
description: Evaluates information retrieval systems on pandemic-related queries using a dynamically evolving corpus. It probes a system's ability to retrieve relevant medical literature under real-world conditions where terminology and document availability change rapidly. Use when the user wants to benchmark on TREC-COVID Round 1, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.04474
  bibtex_key: voorhees2020trecovid
  confidence: high
---

# trec-covid-eval

> TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection — Voorhees et al. (2020) (arXiv:2005.04474, 2020)

## What this evaluates

Evaluates information retrieval systems on pandemic-related queries using a dynamically evolving corpus. It probes a system's ability to retrieve relevant medical literature under real-world conditions where terminology and document availability change rapidly.

## Datasets

- **TREC-COVID Round 1** — total ?; splits: test (30)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by comparing it to an ideal graded relevance ranking, discounting gains logarithmically by position and normalizing by the ideal DCG.

## Input / output format

**Input**: A topic description (query) and a ranked list of candidate documents retrieved by the system.

**Output**: A ranked list of document IDs ordered by predicted relevance.

## Scoring recipe

```python
def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0
# Map 'Partially Relevant' to 1, 'Relevant' to 2, unjudged/other to 0
```

## Common pitfalls

- Relevance is graded ('Partially Relevant' vs 'Relevant'), not binary; treating all judged documents as binary relevant/non-relevant will distort NDCG scores.
- Depth-7 priority-based pooling means unjudged documents are not guaranteed to be non-relevant, violating the standard assumption required for unbiased estimation of some IR metrics.

## Evidence (verbatim from paper)

> Figure 5 provides a view of how effective the participants' systems were as a group. The figure contains a box-and-whiskers plot of the NDCG@10 scores across all 143 submitted runs for each topic.

## Citation

```bibtex
@misc{voorhees2020trecovid,
  title={TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection},
  author={Voorhees et al. (2020)},
  year={2020},
  note={arXiv:2005.04474}
}
```

- arXiv: 2005.04474