Lst Speech Text Bench Eval

---
name: lst-speech-text-bench-eval
description: Evaluates narrative understanding, commonsense reasoning, and topic coherence in speech-text models by selecting the most plausible continuation from multiple candidates. The benchmark tests both speech-to-speech and text-to-text modes to assess cross-modal alignment and reasoning capabilities under compute constraints. Use when the user wants to benchmark on HellaSwag (sHellaSWAG), StoryCloze, TopicStoryCloze, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06195
  bibtex_key: lu2025latentspeechtexttransformer
  confidence: high
---

# lst-speech-text-bench-eval

> Latent Speech-Text Transformer — Lu et al. (arXiv:2510.06195, 2025)

## What this evaluates

Evaluates narrative understanding, commonsense reasoning, and topic coherence in speech-text models by selecting the most plausible continuation from multiple candidates. The benchmark tests both speech-to-speech and text-to-text modes to assess cross-modal alignment and reasoning capabilities under compute constraints.

## Datasets

- **HellaSwag (sHellaSWAG)** — total ?; splits: test (-1)
- **StoryCloze** — total ?; splits: test (-1)
- **TopicStoryCloze** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of instances where the model correctly selects the most plausible continuation from the given multiple-choice candidates.

## Input / output format

**Input**: Narrative context (speech or text) followed by multiple candidate endings (speech or text).

**Output**: Selected candidate index or continuation text.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Speech prompts and responses are generated independently with Kokoro TTS and concatenated, requiring models to handle modality switching correctly.
- Evaluation covers both speech-to-speech and text-to-text modes, so results must be reported separately for each modality.
- Datasets are resynthesized for higher-quality speech, meaning raw speech benchmarks may yield different results.

## Evidence (verbatim from paper)

> We evaluate the model on three benchmarks, where each dataset provides a narrative context and candidate endings, and the model selects the most plausible continuation. Table 2: Evaluation datasets for story completion (MC = Multiple Choice).

## Citation

```bibtex
@misc{lu2025latentspeechtexttransformer,
  title={Latent Speech-Text Transformer},
  author={Lu et al.},
  year={2025},
  note={arXiv:2510.06195}
}
```

- arXiv: 2510.06195