Seniortalk Eval

---
name: seniortalk-eval
description: Evaluates speech processing models on authentic, real-world conversations among super-aged Chinese speakers (75+). It probes capabilities in speaker verification, diarization, automatic speech recognition, and speech editing under conditions of age-related vocal degradation, dialectal variation, and presbyphonia. Use when the user wants to benchmark on SeniorTalk, or asks about evaluating this task. Reports EER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.16578
  bibtex_key: chen2025seniortalk
  confidence: high
---

# seniortalk-eval

> SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors — Yang Chen et al. (2025) (arXiv:2503.16578, 2025)

## What this evaluates

Evaluates speech processing models on authentic, real-world conversations among super-aged Chinese speakers (75+). It probes capabilities in speaker verification, diarization, automatic speech recognition, and speech editing under conditions of age-related vocal degradation, dialectal variation, and presbyphonia.

## Datasets

- **SeniorTalk** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `EER` **(primary)** — range: percent
  - Equal Error Rate: the threshold τ where the miss probability equals the false alarm probability. The system accepts speakers as the same if similarity > τ, rejects if < τ.
- `minDCF` — range: percent
  - Minimum Detection Cost: a cost-sensitive metric evaluating speaker verification systems under application-specific conditions.
- `DER` — range: percent
  - Diarization Error Rate: DER = (FA + MD + Conf) / T, where FA is false alarm segments, MD is missed detection segments, Conf is confusion segments, and T is total reference speech segments.
- `CER` — range: percent
  - Character Error Rate: CER = (S + D + I) / N, where S, D, I are substitutions, deletions, and insertions, and N is the total number of characters in the reference text.

## Input / output format

**Input**: Raw audio recordings of natural conversations among super-aged speakers, accompanied by reference transcriptions and speaker annotations.

**Output**: For verification/diarization: speaker identity labels or segment boundaries. For ASR: character-level transcriptions. For editing: modified audio waveforms.

## Scoring recipe

```python
def compute_cer(predictions, references):
    total_errors = 0
    total_chars = 0
    for pred, ref in zip(predictions, references):
        # Compute edit distance (substitutions, deletions, insertions)
        edits = levenshtein_distance(pred, ref)
        total_errors += edits
        total_chars += len(ref)
    return (total_errors / total_chars) * 100 if total_chars > 0 else 0.0
```

## Common pitfalls

- Age-related vocal degradation (pitch instability, articulatory imprecision) significantly impacts model performance compared to standard adult speech benchmarks.
- Dialectal variation and regional accents cause substantial CER increases, with heavy accents pushing errors above 80% for some models.
- Gender imbalance (1:3 female-to-male ratio) in the dataset reduces the saliency of secondary sexual voice characteristics, complicating speaker separation.

## Evidence (verbatim from paper)

> We adopt two scoring approaches: probabilistic linear discriminant analysis (PLDA) and cosine similarity, with evaluation based on two metrics: (1) Equal Error Rate (EER): We define a threshold τ where the miss probability equals the false alarm probability. Specifically, if the similarity score is above this threshold, the system accepts that the speakers are the same person; if it is below this threshold, the system rejects the claim. This threshold is selected when the false acceptance rate equals the false rejection rate. (2) Minimum Detection Cost (minDCF): A cost-sensitive metric for evaluating speaker verification systems under application-specific conditions.

## Citation

```bibtex
@misc{chen2025seniortalk,
  title={SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors},
  author={Yang Chen et al. (2025)},
  year={2025},
  note={arXiv:2503.16578}
}
```

- arXiv: 2503.16578