Crossnews Ua Eval

---
name: crossnews-ua-eval
description: Evaluates cross-lingual semantic similarity between news article pairs across four dimensions (Who, What, Where, When) in Ukrainian, Polish, Russian, and English. It probes a model's ability to align event-level information across languages while ignoring publication dates. Use when the user wants to benchmark on CrossNews-UA, or asks about evaluating this task. Reports macro-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.19628
  bibtex_key: dementieva2025crossnewsua
  confidence: high
---

# crossnews-ua-eval

> CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English — Dementieva et al. (2025) (arXiv:2510.19628, 2025)

## What this evaluates

Evaluates cross-lingual semantic similarity between news article pairs across four dimensions (Who, What, Where, When) in Ukrainian, Polish, Russian, and English. It probes a model's ability to align event-level information across languages while ignoring publication dates.

## Datasets

- **CrossNews-UA** — total ?; splits: test (-1); repo https://github.com/TUM-NLP/crossnews-ua

## Metrics

- `macro-averaged F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall computed per similarity label, then averaged macroscopically across the four 4W dimensions (Who, What, Where, When) and language pairs.

## Input / output format

**Input**: Pairs of news article texts in two different languages. Publication dates are explicitly excluded from the input.

**Output**: A similarity label for each of the four dimensions: Who, What, Where, When.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    dims = ["Who", "What", "Where", "When"]
    f1_scores = []
    for dim in dims:
        pred_labels = [p[dim] for p in predictions]
        gold_labels = [g[dim] for g in gold]
        f1_scores.append(f1_score(gold_labels, pred_labels, average="binary"))
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models must not use publication dates as a proxy for temporal similarity, as annotators were explicitly instructed to exclude them.
- Performance varies significantly across the 4W dimensions, with 'When' consistently being the hardest, so reporting a single aggregate without dimension breakdown can be misleading.

## Evidence (verbatim from paper)

> *Табл. 3: The detailed results for e5-large embeddings baseline per each language pair and dimension (macro-averaged F1-score).*

## Citation

```bibtex
@misc{dementieva2025crossnewsua,
  title={CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English},
  author={Dementieva et al. (2025)},
  year={2025},
  note={arXiv:2510.19628}
}
```

- arXiv: 2510.19628