Simultaneous S2st Eval

---
name: simultaneous-s2st-eval
description: Evaluates simultaneous speech-to-speech translation models on translation accuracy, latency, speaker voice preservation, and audio naturalness across multiple languages. It probes the model's ability to generate high-quality target speech in real-time without relying on word-level aligned training data. Use when the user wants to benchmark on Audio-NTREX-4L, Europarl-ST, or asks about evaluating this task. Reports ASR-BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.11072
  bibtex_key: [REDACTED]
  confidence: high
---

# simultaneous-s2st-eval

> Simultaneous Speech-to-Speech Translation Without Aligned Data — Labiausse et al. (2026) (arXiv:2602.11072, 2026)

## What this evaluates

Evaluates simultaneous speech-to-speech translation models on translation accuracy, latency, speaker voice preservation, and audio naturalness across multiple languages. It probes the model's ability to generate high-quality target speech in real-time without relying on word-level aligned training data.

## Datasets

- **Audio-NTREX-4L** — total 1200; splits: valid (600), test (600)
- **Europarl-ST** — total 8192; splits: valid (4096), test (4096)

## Metrics

- `ASR-BLEU` **(primary)** — range: [0, 100]
  - BLEU score computed on Whisper-medium transcribed generated speech versus the reference translation, after normalizing both hypothesis and reference texts to reduce ASR error impact.
- `LAAL` — range: seconds
  - Length-Adaptive Average Lagging, approximating the average time (seconds) between a source word and its translation. Computed as (1/n_max) * sum(d_i - (i-1)*gamma) for i=1 to n_max, where gamma = source_duration / max(n_gen, n_ref) and n_max is the index where emission time d_i >= source_duration.
- `Speaker Similarity` — range: [-1, 1]
  - Cosine similarity between the source and generated speech embeddings using the WavLM Large model. Values in tables are scaled by 100.
- `End Offset` — range: seconds
  - Time difference in seconds between the end of the last generated word and the end of the last word from the source utterance.
- `ASR-COMET` — range: [0, 1]
  - COMET score (using XCOMET-XL) computed on Whisper-medium transcribed generated speech versus the reference translation.

## Input / output format

**Input**: Multilingual source speech audio (French, Spanish, Portuguese, German, or Italian).

**Output**: Target English speech audio and a corresponding text stream.

## Scoring recipe

```python
def compute_metrics(generated_audio, source_audio, reference_text):
    # ASR-BLEU
    hyp_text = normalize(whisper.transcribe(generated_audio, model='medium'))
    ref_text = normalize(reference_text)
    asr_bleu = compute_bleu(hyp_text, ref_text)
    # LAAL
    timestamps = whisper.get_word_timestamps(generated_audio)
    source_dur = get_duration(source_audio)
    n_gen = len(timestamps)
    n_ref = len(reference_text.split())
    gamma = source_dur / max(n_gen, n_ref)
    n_max = next(i for i, t in enumerate(timestamps) if t >= source_dur)
    laal = sum(timestamps[i] - (i-1)*gamma for i in range(1, n_max+1)) / n_max
    return asr_bleu, laal
```

## Common pitfalls

- ASR-BLEU scores are heavily influenced by Whisper's transcription accuracy; failing to normalize hypothesis and reference texts as specified will inflate error rates.
- LAAL requires precise word-level emission timestamps from the ASR model; using a different ASR model or varying sampling rates will break the formula's assumptions.
- Speaker similarity values in the tables are scaled by 100 (cosine similarity × 100), which may confuse readers expecting the standard [-1, 1] range.

## Evidence (verbatim from paper)

> We evaluate translation quality by transcribing generated speech using Whisper medium (Radford et al., 2023) and computing BLEU (Post, 2018) and COMET (Rei et al., 2020) scores with respect to a reference translation, referred to as ASR-BLEU and ASR-COMET. To reduce the impact of ASR errors, hypothesis and reference texts are normalized before computing BLEU scores.

## Citation

```bibtex
@misc{labiausse2026simultaneousspeech,
  title={Simultaneous Speech-to-Speech Translation Without Aligned Data},
  author={Labiausse et al. (2026)},
  year={2026},
  note={arXiv:2602.11072}
}
```

- arXiv: 2602.11072