Xmind Crosslingual Eval

---
name: xmind-crosslingual-eval
description: This benchmark evaluates the cross-lingual transfer capability of neural news recommenders. It probes how well models trained monolingually on English news can generate accurate recommendations in 14 other languages under zero-shot and few-shot settings, with and without bilingual user consumption patterns. Use when the user wants to benchmark on xMIND, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.17876
  bibtex_key: iana2024mindyourlanguage
  confidence: high
---

# xmind-crosslingual-eval

> MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation — Iana et al. (2024) (arXiv:2403.17876, 2024)

## What this evaluates

This benchmark evaluates the cross-lingual transfer capability of neural news recommenders. It probes how well models trained monolingually on English news can generate accurate recommendations in 14 other languages under zero-shot and few-shot settings, with and without bilingual user consumption patterns.

## Datasets

- **xMIND** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/andreeaiana/xMIND

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, calculated as the average of the reciprocal of the rank of the first relevant item in the recommended list.
- `nDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 5, measuring ranking quality by discounting gains logarithmically based on position.
- `nDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 10, measuring ranking quality by discounting gains logarithmically based on position.

## Input / output format

**Input**: User click history (up to 50 previously clicked news items with text, categories, and named entities) and a set of candidate news items (text, categories, named entities) to be scored.

**Output**: A ranked list of candidate news items, or a scalar recommendation score per candidate computed as the dot product between the user history representation and the candidate news representation.

## Scoring recipe

```python
import math

def compute_metrics(predictions, gold):
    # predictions: list of (score, item_id) sorted descending
    # gold: set of relevant item_ids
    pos = [s for s, _ in predictions if _ in gold]
    neg = [s for s, _ in predictions if _ not in gold]
    auc = sum(1 for p in pos for n in neg if p > n) / (len(pos) * len(neg)) if pos and neg else 0.0
    mrr = 0.0
    for rank, (_, item_id) in enumerate(predictions, 1):
        if item_id in gold:
            mrr = 1.0 / rank
            break
    k = 10
    dcg = sum(1.0 / math.log2(r + 1) for r, (_, item_id) in enumerate(predictions[:k], 1) if item_id in gold)
    idcg = sum(1.0 / math.log2(r + 1) for r in range(1, min(len(gold), k) + 1))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return {'AUC': auc, 'MRR': mrr, 'nDCG@10': ndcg}
```

## Common pitfalls

- The test set uses the validation portion of MIND because official test labels were not released, which may not reflect true out-of-distribution performance.
- Hyperparameter tuning is performed exclusively on the English (MIND) dataset, potentially introducing distribution bias that does not generalize to target languages.
- Bilingual consumption is simulated by randomly replacing English news with translated versions, which may not capture real-world bilingual user behavior or language mixing patterns.

## Evidence (verbatim from paper)

> We repeat each experiment three times, with different random seeds, and report averages and standard deviations for the standard metrics: AUC, MRR, nDCG@5, and nDCG@10.

## Citation

```bibtex
@misc{iana2024mindyourlanguage,
  title={MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation},
  author={Iana et al. (2024)},
  year={2024},
  note={arXiv:2403.17876}
}
```

- arXiv: 2403.17876