Multilingual Tot Sim Eval

---
name: multilingual-tot-sim-eval
description: Evaluates the fidelity of synthetic Tip-of-the-Tongue (ToT) queries by measuring how well they reproduce the relative ranking of retrieval systems compared to real human-authored ToT queries across four languages. Use when the user wants to benchmark on Multilingual ToT Test Collection, or asks about evaluating this task. Reports Kendall's tau & Pearson's r.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21096
  bibtex_key: [REDACTED]
  confidence: high
---

# multilingual-tot-sim-eval

> Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation — He et al. (2026) (arXiv:2604.21096, 2026)

## What this evaluates

Evaluates the fidelity of synthetic Tip-of-the-Tongue (ToT) queries by measuring how well they reproduce the relative ranking of retrieval systems compared to real human-authored ToT queries across four languages.

## Datasets

- **Multilingual ToT Test Collection** — total 20000; splits: synthetic (20000), validation_real (600); repo https://github.com/kimdanny/ntcir-19-tot

## Metrics

- `Kendall's tau & Pearson's r` **(primary)** — range: [-1, 1]
  - Computes the rank correlation between the system rankings induced by real human ToT queries ($Q_{real}$) and synthetic ToT queries ($Q_{syn}$). Kendall's $	au$ measures the proportion of concordant versus discordant pairs, while Pearson's $r$ measures linear correlation between the rank positions.

## Input / output format

**Input**: Wikipedia page text (or LLM-generated summary) and a language-specific prompt template instructing the model to simulate a Tip-of-the-Tongue query for a given entity.

**Output**: A natural language query in the target language describing the entity, explicitly avoiding the entity's name.

## Scoring recipe

```python
# rankings_real: list of system scores/ranks from Q_real
# rankings_syn: list of system scores/ranks from Q_syn
tau = kendalltau(rankings_real, rankings_syn)
r = pearsonr(rankings_real, rankings_syn)
return tau, r
```

## Common pitfalls

- Correlation measures rank order agreement, not absolute metric values (e.g., NDCG scores can differ widely while correlation remains high).
- The validation set for non-English languages includes machine-translated queries, which may introduce noise compared to purely human-authored queries.
- Post-generation anonymity checks discard queries that reveal the entity name, potentially biasing the final set toward entities with more distinctive descriptive features.

## Evidence (verbatim from paper)

> To quantify the agreement between the two rankings, we compute both Kendall’s $	au$ and Pearson’s $r$ correlation coefficients between $R_{	ext{real}}$ and $R_{	ext{syn}}$. A high correlation indicates that the synthetic queries preserve the relative ordering of retrieval systems observed under real user queries, thereby validating the quality of the simulated ToT queries.

## Citation

```bibtex
@misc{he2026multilingual,
  title={Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2604.21096}
}
```

- arXiv: 2604.21096