Longspeech Eval

---
name: longspeech-eval
description: Evaluates long-form speech processing capabilities across transcription, translation, summarization, and higher-level reasoning tasks. It probes models' ability to maintain semantic consistency, track temporal progression, and extract structured information from ~10-minute audio segments. Use when the user wants to benchmark on LongSpeech, or asks about evaluating this task. Reports WER, BLEU-4, Numeric Accuracy, Strict Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.13539
  bibtex_key: yang2026longspeech
  confidence: high
---

# longspeech-eval

> LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech — Fei Yang et al. (2026) (arXiv:2601.13539, 2026)

## What this evaluates

Evaluates long-form speech processing capabilities across transcription, translation, summarization, and higher-level reasoning tasks. It probes models' ability to maintain semantic consistency, track temporal progression, and extract structured information from ~10-minute audio segments.

## Datasets

- **LongSpeech** — total 100000; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: [0, 1]
  - Word Error Rate: the ratio of word-level edit errors (insertions, deletions, substitutions) to total words in the reference transcription. Lower values indicate better performance.
- `BLEU-4` **(primary)** — range: percent
  - n-gram precision with brevity penalty, computed case-insensitively. Higher scores indicate better translation fluency and adequacy.
- `ROUGE-1/2/L F1` — range: [0, 1]
  - Recall-oriented n-gram overlap metrics measuring unigram, bigram, and longest common subsequence F1 scores. Higher values indicate better content coverage and coherence.
- `Numeric Accuracy` **(primary)** — range: [0, 1]
  - Proportion of responses that exactly match the ground truth numeric value.
- `Parsability Rate` — range: [0, 1]
  - Fraction of inputs where the model successfully parses the query structure (e.g., identifies target entity and operation).
- `Post-Parsing Precision` — range: [0, 1]
  - Accuracy of answers among successfully parsed queries.
- `Misunderstanding Rate` — range: [0, 1]
  - Proportion of cases where the model misinterprets the question intent (e.g., confuses entities or operations).
- `Strict Accuracy` **(primary)** — range: [0, 1]
  - Percentage of predictions matching the coarse label exactly, or ratio of fully correct 'YES' judgments for temporal localization.
- `Relaxed Accuracy` — range: [0, 1]
  - Rate at which the predicted label shares the same broad category/polarity, or ratio of 'YES' or 'PARTIALLY' judgments.
- `Detection Accuracy` — range: [0, 1]
  - Accuracy of language detection predictions.
- `Detection Errors` — range: [0, 1]
  - Error rate for language detection predictions.

## Input / output format

**Input**: Audio segment (~10 minutes) paired with a task-specific text prompt (e.g., transcription instruction, translation query, summarization request, or QA question).

**Output**: Text response corresponding to the task: raw transcription, translated text, summary, numeric value, structured extraction, emotion category label, or temporal localization coordinates.

## Scoring recipe

```python
def score(predictions, golds, task):
    if task == 'ASR':
        return wer(predictions, golds)  # edit_dist / len(golds)
    elif task == 'S2TT':
        return bleu(predictions, golds)  # case-insensitive BLEU-4
    elif task == 'Summarization':
        return rouge_f1(predictions, golds, n=1,2,L)
    elif task in ['Content Separation', 'Speaker Count']:
        parsable = parse_query(predictions)
        num_acc = exact_match(predictions, golds)
        post_prec = accuracy(predictions[golds], parsable)
        mis_rate = 1 - parsability_rate(predictions)
        return num_acc, parsable, post_prec, mis_rate
    elif task == 'Emotion':
        coarse_pred = map_to_coarse(predictions)
        strict = exact_match(coarse_pred, golds)
        relaxed = match_polarity(coarse_pred, golds)
        return strict, relaxed
    elif task == 'Temporal':
        judgments = gpt4_judge(predictions, golds)  # YES/NO/PARTIALLY
        strict = (judgments == 'YES').mean()
        relaxed = (judgments in ['YES', 'PARTIALLY']).mean()
        return strict, relaxed
```

## Common pitfalls

- Models may correctly parse the query intent but fail to extract the precise answer (e.g., high parsability rate but low numeric accuracy).
- Models lacking native long-audio support may output placeholder tokens (e.g., '[music]') instead of actual transcriptions, requiring segmentation-based evaluation pipelines.
- Emotion analysis requires mapping fine-grained model outputs to 7 predefined coarse categories before computing strict/relaxed accuracy.

## Evidence (verbatim from paper)

> We evaluate speech recognition performance using: Word Error Rate (WER): the ratio of word-level edit errors (insertions, deletions, substitutions) to total words in the reference transcription. Lower values indicate better performance. Speech-to-Text Translation For end-to-end translation from speech to text, we use BLEU: n-gram precision with brevity penalty, computed case-insensitively as BLEU-4. Higher scores indicate better translation fluency and adequacy.

## Citation

```bibtex
@misc{yang2026longspeech,
  title={LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech},
  author={Fei Yang et al. (2026)},
  year={2026},
  note={arXiv:2601.13539}
}
```

- arXiv: 2601.13539