Sentence Stress Detection Eval

---
name: sentence-stress-detection-eval
description: Probes a model's ability to detect sentence-level prosodic stress at the word/token level using only audio input. It evaluates zero-shot generalization and alignment-free stress identification across diverse speech styles and synthetic/human datasets. Use when the user wants to benchmark on TinyStress-15K, Aix-MARSEC, Expresso, EmphAssess, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19103
  bibtex_key: yosha2025whistress
  confidence: high
---

# sentence-stress-detection-eval

> WHISTRESS: Enriching Transcriptions with Sentence Stress Detection — Yosha et al. (2025) (arXiv:2505.19103, 2025)

## What this evaluates

Probes a model's ability to detect sentence-level prosodic stress at the word/token level using only audio input. It evaluates zero-shot generalization and alignment-free stress identification across diverse speech styles and synthetic/human datasets.

## Datasets

- **TinyStress-15K** — total 15000; splits: train (-1), val (-1)
- **Aix-MARSEC** — total 2400; splits: train (1680), test (720)
- **Expresso** — total ?; splits: test (-1)
- **EmphAssess** — total 3652; splits: train (1500), test (2152)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Standard binary classification F1 score: 2 * (Precision * Recall) / (Precision + Recall). Precision and recall are computed at the word level, where a word is labeled stressed if at least one of its tokens is predicted as stressed.

## Input / output format

**Input**: Raw audio recordings. Transcriptions are not required during inference.

**Output**: Binary stress label per token, aggregated to word-level if at least one token is marked stressed.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- Confusing token-level predictions with word-level labels: the paper explicitly aggregates token predictions to word-level by marking a word stressed if at least one of its tokens is predicted as stressed.
- Assuming forced alignment or timestamps are required for inference: the model is explicitly alignment-free; baselines use MFA or ground-truth timestamps, but WhiStress does not.
- Mixing up zero-shot vs in-domain evaluation splits: zero-shot uses TinyStress-15K for training and evaluates on Expresso/EmphAssess; in-domain training on EmphAssess uses speakers ex03/ex04 for training and ex01/ex02 for testing.

## Evidence (verbatim from paper)

> We report performance using standard classification metrics: precision, recall and F$1$ score. In our settings, a word is considered stressed by the WhiStress model if at least one of its tokens is marked as stressed.

## Citation

```bibtex
@misc{yosha2025whistress,
  title={WHISTRESS: Enriching Transcriptions with Sentence Stress Detection},
  author={Yosha et al. (2025)},
  year={2025},
  note={arXiv:2505.19103}
}
```

- arXiv: 2505.19103