Trec2022 Neuclir Eval

---
name: trec2022-neuclir-eval
description: Evaluates neural cross-language information retrieval systems on ad hoc, reranking, and monolingual tasks across Chinese, Persian, and Russian newswire collections. It measures how well models retrieve and rank relevant documents when queries are in English and documents are in other languages, or when queries are human-translated. Use when the user wants to benchmark on TREC 2022 NeuCLIR Collections, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.12367
  bibtex_key: lawrie2023overview
  confidence: high
---

# trec2022-neuclir-eval

> Overview of the TREC 2022 NeuCLIR Track — Dawn Lawrie et al. (2023) (arXiv:2304.12367, 2023)

## What this evaluates

Evaluates neural cross-language information retrieval systems on ad hoc, reranking, and monolingual tasks across Chinese, Persian, and Russian newswire collections. It measures how well models retrieve and rank relevant documents when queries are in English and documents are in other languages, or when queries are human-translated.

## Datasets

- **TREC 2022 NeuCLIR Collections** — total ?; splits: test (-1); repo https://github.com/NeuCLIR/download-collection

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Standard TREC IR metric calculated as DCG@10 / IDCG@10, where DCG sums relevance scores discounted by log2(rank+1).

## Input / output format

**Input**: English topics (or human-translated target-language topics for monolingual task) paired with document collections in Chinese, Persian, or Russian. For reranking, an initial BM25-ranked list of 1,000 documents is provided.

**Output**: A ranked list of 1,000 document IDs ordered by predicted relevance.

## Scoring recipe

```python
def compute_ndcg_at_10(predictions, gold):
    dcg = 0.0
    idcg = 0.0
    for i, doc_id in enumerate(predictions[:10]):
        rel = gold.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    sorted_rels = sorted(gold.values(), reverse=True)[:10]
    for i, rel in enumerate(sorted_rels):
        idcg += (2**rel - 1) / math.log2(i + 2)
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Reranking task uses a fixed initial run of 1,000 documents, not the full collection.
- Monolingual tasks require human-translated topics, not machine-translated ones.
- Runs using human-in-the-loop processes must be explicitly flagged as 'manual'.

## Evidence (verbatim from paper)

> Systems are then asked to rerank the documents to produce a new ordering that improves an evaluation metric. This task is suitable for teams that want to focus on second-stage scoring models, rather than on models which search an entire collection.

## Citation

```bibtex
@misc{lawrie2023overview,
  title={Overview of the TREC 2022 NeuCLIR Track},
  author={Dawn Lawrie et al. (2023)},
  year={2023},
  note={arXiv:2304.12367}
}
```

- arXiv: 2304.12367