Discourse Stress Tts Eval

---
name: discourse-stress-tts-eval
description: Evaluates whether text-to-speech systems can correctly realize discourse-dependent word-level stress based on contrasting contexts. It probes the model's ability to adapt prosodic emphasis dynamically rather than relying on fixed sentence-internal stress patterns. Use when the user wants to benchmark on CAST, or asks about evaluating this task. Reports Pair-Correct.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10580
  bibtex_key: [REDACTED]
  confidence: high
---

# discourse-stress-tts-eval

> Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark — Turetzky et al. (2026) (arXiv:2604.10580, 2026)

## What this evaluates

Evaluates whether text-to-speech systems can correctly realize discourse-dependent word-level stress based on contrasting contexts. It probes the model's ability to adapt prosodic emphasis dynamically rather than relying on fixed sentence-internal stress patterns.

## Datasets

- **CAST** — total 10000; splits: test (10000)

## Metrics

- `Hit` — range: [0, 1]
  - 1[A ∈ Ŷ_A] where A is the intended stressed word and Ŷ_A is the set of detected stressed words in the synthesized audio. Measures basic stress expressiveness regardless of additional stressed words.
- `Pair-Contrast` — range: [0, 1]
  - 1[A ∈ Ŷ_A ∧ B ∉ Ŷ_A] requiring the correct stressed word to be present while the alternative context's stressed word is absent. Penalizes over-expressive stress.
- `Pair-Correct` **(primary)** — range: [0, 1]
  - 1[(A ∈ Ŷ_A ∧ B ∉ Ŷ_A) ∧ (B ∈ Ŷ_B ∧ A ∉ Ŷ_B)] requiring Pair-Contrast to hold for both sides of a contrastive context pair simultaneously. The strictest metric, ensuring genuine context adaptation.

## Input / output format

**Input**: Target sentence paired with a contrasting context, provided via conditioning modes (no context, prepended context, natural language instruction, or explicit stress markup).

**Output**: Synthesized audio waveform for the target sentence.

## Scoring recipe

```python
# predictions: detected stressed words per sample (Ŷ_A, Ŷ_B)
# gold: intended stressed words per sample (A, B)
hit = 1.0 if A in Ŷ_A else 0.0
pair_contrast = 1.0 if (A in Ŷ_A) and (B not in Ŷ_A) else 0.0
pair_correct = 1.0 if (pair_contrast and (B in Ŷ_B) and (A not in Ŷ_B)) else 0.0
return hit, pair_contrast, pair_correct
```

## Common pitfalls

- Systems with a sentence-internal bias toward stressing a specific word may artificially satisfy Pair-Contrast for one side of a pair by chance.
- Over-expressive stress (stressing multiple words) will cause a system to fail Pair-Contrast even if the target word is correctly emphasized.
- The automatic stress detector (WHISTRESS) is used as a proxy for human perception, and its moderate agreement (κ=0.32) means metric scores may not perfectly align with human judgments.

## Evidence (verbatim from paper)

> Pair-Correct (Correct): a pair-level metric requiring Pair Contrast to hold for both sides of the contrastive context pair: $$ \mathbf {1} \left[ \left(A \in \hat {Y} _ {A} \wedge B \notin \hat {Y} _ {A}\right) \wedge \left(B \in \hat {Y} _ {B} \wedge A \notin \hat {Y} _ {B}\right)\right]. $$ This is the strictest metric, requiring Pair Contrast to hold for both sides simultaneously.

## Citation

```bibtex
@misc{turetzky2026knowing,
  title={Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark},
  author={Turetzky et al. (2026)},
  year={2026},
  note={arXiv:2604.10580}
}
```

- arXiv: 2604.10580