Crosslingual Speech Text Retrieval Eval

---
name: crosslingual-speech-text-retrieval-eval
description: Evaluates cross-lingual speech-to-text retrieval and intent detection capabilities across multiple datasets, testing how well speech queries can retrieve relevant text documents or classify intents without intermediate ASR or translation pipelines. Use when the user wants to benchmark on Kallaama-Retrieval-Eval, Fleurs-Retrieval-Eval, Urban Bus, WolBanking77, or asks about evaluating this task. Reports nDCG@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.19991
  bibtex_key: [REDACTED]
  confidence: high
---

# crosslingual-speech-text-retrieval-eval

> Cross-lingual Matryoshka Representation Learning across Speech and Text — Sy et al. (2026) (arXiv:2602.19991, 2026)

## What this evaluates

Evaluates cross-lingual speech-to-text retrieval and intent detection capabilities across multiple datasets, testing how well speech queries can retrieve relevant text documents or classify intents without intermediate ASR or translation pipelines.

## Datasets

- **Kallaama-Retrieval-Eval** — total ?; splits: (unstated)
- **Fleurs-Retrieval-Eval** — total ?; splits: (unstated)
- **Urban Bus** — total ?; splits: test (-1)
- **WolBanking77** — total ?; splits: (unstated)

## Metrics

- `nDCG@5` **(primary)** — range: percent
  - Normalized Discounted Cumulative Gain at k=5. Measures ranking quality by comparing the graded relevance of the top 5 retrieved documents to an ideal ranking.
- `nDCG@10` — range: percent
  - Normalized Discounted Cumulative Gain at k=10. Same as nDCG@5 but evaluates the top 10 retrieved documents.
- `F1-Score` — range: percent
  - Harmonic mean of precision and recall for intent detection and keyword spotting tasks.
- `Recall` — range: percent
  - Proportion of correctly identified positive instances (keywords or intents) out of all actual positives.

## Input / output format

**Input**: Speech audio query (and optionally text query for dual-encoder baselines) paired with a corpus of text documents for retrieval; or speech audio query for intent/keyword spotting.

**Output**: Ranked list of retrieved text documents (for retrieval tasks); predicted intent label or keyword match (for spotting/detection tasks).

## Scoring recipe

```python
def compute_ndcg_at_k(retrieved, gold, k):
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(retrieved[:k]))
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(sorted(gold, reverse=True)[:k]))
    return (dcg / idcg) * 100 if idcg > 0 else 0.0

def compute_f1_recall(preds, golds):
    tp = sum(p == g for p, g in zip(preds, golds))
    fp = sum(p != g for p in preds if p not in golds)
    fn = sum(g != p for g in golds if g not in preds)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return f1 * 100, rec * 100
```

## Common pitfalls

- Matryoshka embedding dimensionality significantly impacts performance, with lower dimensions degrading faster on low-quality speech (Fleurs) than high-quality speech (Kallaama).
- Pipelined baselines (ASR then text retrieval) suffer from transcription error propagation, which the direct speech-text models avoid.
- NLLB-LLM2Vec baseline uses a fixed 4096-dimension embedding, making direct comparison with variable-dimension Matryoshka models require careful normalization or dimension-matching.

## Evidence (verbatim from paper)

> We evaluate the trained models on Kallaama-Retrieval-Eval and Fleurs-Retrieval-Eval using nDCG, a standard metric to evaluate the ranking of recommender systems. nDCG@$k$ measures the ranking quality of the top $k$ retrieved documents by comparing their graded relevance to an ideal ranking, with higher scores indicating better alignment with the ground truth.

## Citation

```bibtex
@misc{sy2026crosslingualmatryoshka,
  title={Cross-lingual Matryoshka Representation Learning across Speech and Text},
  author={Sy et al. (2026)},
  year={2026},
  note={arXiv:2602.19991}
}
```

- arXiv: 2602.19991