Wilder Eval

---
name: wilder-eval
description: This benchmark evaluates automatic speech recognition (ASR) capabilities on Mandarin speech produced by elderly individuals. It probes a model's robustness to real-world acoustic degradation, articulation variability, tremors, and diverse accent strengths under uncontrolled recording conditions. Use when the user wants to benchmark on WildElder, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.09344
  bibtex_key: wang2025wilder
  confidence: medium
---

# wilder-eval

> WildElder: A Chinese Elderly Speech Dataset from the Wild with Fine-Grained Manual Annotations — Wang et al. (2025) (arXiv:2510.09344, 2025)

## What this evaluates

This benchmark evaluates automatic speech recognition (ASR) capabilities on Mandarin speech produced by elderly individuals. It probes a model's robustness to real-world acoustic degradation, articulation variability, tremors, and diverse accent strengths under uncontrolled recording conditions.

## Datasets

- **WildElder** — total 23701; splits: train (18835), dev (2465), test (2400); repo https://github.com/NKU-HLT/WildElder

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Standard ASR metric measuring the percentage of words incorrectly recognized relative to the ground truth transcript. Calculated as (Substitutions + Deletions + Insertions) / Total Words in reference.

## Input / output format

**Input**: Raw audio recordings of Mandarin speech (average ~5.1 seconds per utterance) collected from online videos.

**Output**: Predicted text transcript corresponding to each input audio utterance.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = 0
    total_errors = 0
    for pred, ref in zip(predictions, references):
        ref_words = ref.split()
        pred_words = pred.split()
        total_words += len(ref_words)
        edits = levenshtein_distance(pred_words, ref_words)
        total_errors += edits
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- Splits are performed at the speaker level, not utterance level, so standard random splitting will cause severe data leakage.
- The dataset contains 'in-the-wild' recordings with uncontrolled background noise, tremors, and varying accent strengths, making lab-clean baselines misleading.
- Mandarin speech is character-based; using word-level tokenization without proper Chinese word segmentation will inflate error rates.

## Evidence (verbatim from paper)

> To ensure reproducibility, WildElder is divided into training, development, and test sets at the speaker level. The training set contains 18,835 utterances (26.7 h), the development set 2,465 utterances (3.5 h), and the test set 2,400 utterances (3.5 h). The average utterance duration is around 5.1 seconds across all subsets, showing a balanced partition suitable for training and evaluation.

## Citation

```bibtex
@misc{wang2025wilder,
  title={WildElder: A Chinese Elderly Speech Dataset from the Wild with Fine-Grained Manual Annotations},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.09344}
}
```

- arXiv: 2510.09344