Liveclktb Eval

---
name: liveclktb-eval
description: Evaluates multilingual LLMs on their ability to transfer factual knowledge across languages using time-sensitive, real-world events that occur after the model's training cutoff. It measures both in-language factual recall and cross-lingual generalization performance across domains like music, movies, and sports. Use when the user wants to benchmark on LiveCLKTBench, or asks about evaluating this task. Reports Transfer Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.14774
  bibtex_key: [REDACTED]
  confidence: medium
---

# liveclktb-eval

> LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs — Guo et al. (2025) (arXiv:2511.14774, 2025)

## What this evaluates

Evaluates multilingual LLMs on their ability to transfer factual knowledge across languages using time-sensitive, real-world events that occur after the model's training cutoff. It measures both in-language factual recall and cross-lingual generalization performance across domains like music, movies, and sports.

## Datasets

- **LiveCLKTBench** — total ?; splits: test (-1)

## Metrics

- `Overall Score` — range: [0, 1]
  - Accuracy of model-generated answers against gold answers when evaluated in the source language.
- `Transfer Score` **(primary)** — range: [0, 1]
  - Accuracy of model-generated answers against gold answers when evaluated across different source-target language pairs to measure cross-lingual generalization.

## Input / output format

**Input**: Factual question-answer pairs grounded in source documents, presented in a target language for cross-lingual evaluation.

**Output**: Model-generated text answers to the questions.

## Scoring recipe

```python
def compute_scores(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    overall_score = correct / len(golds)
    # Transfer score aggregates accuracy specifically for cross-lingual instances
    transfer_score = correct / len(golds)  # Computed on cross-lingual subset
    return overall_score, transfer_score
```

## Common pitfalls

- Assuming cross-lingual transfer performance is symmetric across language pairs, whereas the benchmark shows strong directional asymmetry (e.g., en->ja vs ja->en).
- Overestimating the benefits of model scaling, as gains show diminishing returns and saturate at higher parameter counts.
- Ignoring domain-specific variance, as sports consistently yields the weakest performance compared to music and movies.

## Evidence (verbatim from paper)

> Table[2] presents the Overall and Transfer scores for each model across the three domains—music, movies, and sports—as well as their domain averages. Across models, the average Overall Score ranges from 0.221 (OLMo-2-7B) to 0.414 (Gemma-2-9B), while the Transfer Score ranges from 0.567 (OLMo-2-7B) to 0.747 (Qwen2.5-7B).

## Citation

```bibtex
@misc{guo2025liveclktb,
  title={LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2511.14774}
}
```

- arXiv: 2511.14774