Realsi Sst Eval

---
name: realsi-sst-eval
description: Evaluates end-to-end simultaneous speech-to-speech translation quality and latency on long-form, multi-domain continuous speech. Probes the model's ability to maintain semantic accuracy, speaker voice characteristics, and low delay in real-time multilingual dialogue. Use when the user wants to benchmark on RealSI, or asks about evaluating this task. Reports VIP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.17527
  bibtex_key: [REDACTED]
  confidence: high
---

# realsi-sst-eval

> Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice — Cheng et al. (2025) (arXiv:2507.17527, 2025)

## What this evaluates

Evaluates end-to-end simultaneous speech-to-speech translation quality and latency on long-form, multi-domain continuous speech. Probes the model's ability to maintain semantic accuracy, speaker voice characteristics, and low delay in real-time multilingual dialogue.

## Datasets

- **RealSI** — total ?; splits: test (-1)

## Metrics

- `VIP` **(primary)** — range: [0, 100] (score)
  - Human evaluation metric measuring the proportion of valid semantic fragments that accurately convey the speaker’s original intent, closely aligning with human interpreter judgments.
- `SVIP` — range: [0, 100] (score)
  - Speech-specific extension of VIP measuring the proportion of valid speech semantic fragments within a session, requiring core information accuracy, acceptable latency, appropriate pace, and acoustic clarity.
- `BLEURT` — range: [0, 100] or [-1, 1]
  - Automated neural metric for translation quality based on BERT representations.
- `COMET` — range: [0, 100] or [-1, 1]
  - Automated neural metric for translation quality using cross-lingual representations.
- `FLAL` — range: [0, ∞) seconds
  - First Letter Appearance Lagging: time until the system outputs the first determined translation at the paragraph level.
- `AL` — range: [0, ∞) seconds
  - Average Lagging: widely adopted sentence-level latency metric measuring average delay between source and target utterances.
- `LAAL` — range: [0, ∞) seconds
  - Length Adaptive Average Lagging: sentence-level latency metric normalized by output length.

## Input / output format

**Input**: Continuous audio speech in Chinese or English (approx. 5 minutes per sample for long-form; sentence-level for short-form).

**Output**: Translated audio speech in the target language (with voice cloning) or translated text, generated simultaneously as the source audio plays.

## Scoring recipe

```python
def calculate_vip_svip(transcript_segments):
    valid = 0
    total = len(transcript_segments)
    for seg in transcript_segments:
        if (conveys_core_intent(seg) and
            latency_acceptable(seg) and
            pace_acceptable(seg) and
            clarity_acceptable(seg)):
            valid += 1
    return (valid / total) * 100

def calculate_latency(source_timestamps, target_timestamps):
    al = mean(target_timestamps - source_timestamps)
    flal = target_timestamps[0] - source_timestamps[0]
    return al, flal
```

## Common pitfalls

- Automated metrics like BLEURT and COMET may not fully reflect true translation capabilities compared to human VIP/SVIP scores.
- Commercial baselines often employ post-hoc rewriting strategies, making direct latency/quality comparisons with human-like simultaneous interpreters misleading.
- Missing table entries indicate lack of feature support (e.g., speech-to-speech) rather than zero performance.

## Evidence (verbatim from paper)

> For text translation quality assessment, we primarily rely on the idea of the human evaluation metric, Valid Information Proportion (VIP) *[[6]]*, which measures how accurately the translation output conveys the speaker’s original intent for each semantic fragment, closely aligning with human interpreter judgments.

## Citation

```bibtex
@misc{cheng2025seedliveinterpret2,
  title={Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2507.17527}
}
```

- arXiv: 2507.17527