Neuclir 2024 News Retrieval Eval

---
name: neuclir-2024-news-retrieval-eval
description: Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank documents by relevance when queries are in English and documents are in different languages, or when searching across multiple languages simultaneously. Use when the user wants to benchmark on NeuCLIR 1 News Collection, or asks about evaluating this task. Reports nDCG@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.14355
  bibtex_key: lawrie2025neuclir
  confidence: high
---

# neuclir-2024-news-retrieval-eval

> Overview of the TREC 2024 NeuCLIR Track — Dawn Lawrie et al. (arXiv:2509.14355, 2025)

## What this evaluates

Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank documents by relevance when queries are in English and documents are in different languages, or when searching across multiple languages simultaneously.

## Datasets

- **NeuCLIR 1 News Collection** — total ?; splits: test (-1)

## Metrics

- `nDCG@20` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 20. Computed over graded relevance judgments mapped from a 4-point scale to a 3-point scale (3→3, 2→1, 1→0, 0→0). Divides the DCG@20 of the predicted ranking by the ideal DCG@20 based on the gold judgments.

## Input / output format

**Input**: A topic (query) in English, and a document collection in Chinese, Persian, or Russian (for CLIR) or all three combined (for MLIR).

**Output**: A ranked list of up to 1,000 document IDs per topic, ordered by predicted relevance.

## Scoring recipe

```python
def compute_ndcg_at_20(predictions, gold_qrels):
    ndcg_scores = []
    for topic_id, ranked_docs in predictions.items():
        top_20 = ranked_docs[:20]
        dcg = sum(rel / math.log2(i + 2) for i, doc in enumerate(top_20) if (rel := gold_qrels[topic_id].get(doc, 0)) > 0)
        ideal_rels = sorted(gold_qrels[topic_id].values(), reverse=True)[:20]
        idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_rels))
        ndcg_scores.append(dcg / idcg if idcg > 0 else 0.0)
    return sum(ndcg_scores) / len(ndcg_scores)
```

## Common pitfalls

- Relevance judgments use a converted 3-point scale (3→3, 2→1, 1→0, 0→0) from the original 4-point scale; failing to apply this mapping will skew nDCG scores.
- MLIR requires a single unified ranked list across all three language collections, not separate per-language lists.
- Topic filtering rules drop topics with >40% somewhat/very valuable documents or <2 relevant documents, which must be excluded from final metric computation.

## Evidence (verbatim from paper)

> For each topic, the system must return a ranked list of 1,000 documents drawn from the entire target language document collection, ordered by likelihood and degree of relevance to the topic. ... Figure 1. News CLIR nDCG@20. Coordinator runs are marked with slashes. Monolingual runs (i.e., using human-translated topics) are marked with “M” at the top of the bar.

## Citation

```bibtex
@misc{lawrie2025neuclir,
  title={Overview of the TREC 2024 NeuCLIR Track},
  author={Dawn Lawrie et al.},
  year={2025},
  note={arXiv:2509.14355}
}
```

- arXiv: 2509.14355