Taigispeech Eval

---
name: taigispeech-eval
description: Evaluates speech intent recognition in a low-resource, real-world setting for Taiwanese Taigi, specifically testing domain adaptation and robustness to domain mismatch between mined training data and real-world elderly speech. Use when the user wants to benchmark on TaigiSpeech, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.21478
  bibtex_key: chang2026taigispeech
  confidence: medium
---

# taigispeech-eval

> TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild — Kai-Wei Chang et al. (arXiv:2603.21478, 2026)

## What this evaluates

Evaluates speech intent recognition in a low-resource, real-world setting for Taiwanese Taigi, specifically testing domain adaptation and robustness to domain mismatch between mined training data and real-world elderly speech.

## Datasets

- **TaigiSpeech** — total 3079; splits: test (960), fine-tune (1600)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy (correct predictions / total predictions). The exact metric name is not explicitly stated in the provided section, but accuracy is standard for the described classification tasks.

## Input / output format

**Input**: Raw audio utterances (and video clips for the audio-visual mining baseline setup).

**Output**: Discrete class label: 5-class (four emergency intents + Non-Emergency) or binary (Emergency vs. Non-Emergency).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Domain mismatch: Training data comes from mined drama/video clips, while test data is real-world elderly speech, potentially causing performance drops not captured by in-distribution metrics.
- Pseudo-labels treated as ground truth: The drama dataset evaluation uses Gemini-3 pseudo-labels as ground truth, which may overestimate performance if pseudo-labels contain errors.
- Class distribution: The test set is balanced by speaker selection, but real-world deployment may see different intent frequencies.

## Evidence (verbatim from paper)

> To construct a balanced evaluation benchmark, we select 6 speakers from TaigiSpeech to form the test set. The set consists of 3 male and 3 female speakers, with ages distributed to maintain balance across gender and age groups. Each speaker contributes 160 utterances, resulting in a total of 960 test samples. ... Models trained on the mined data are evaluated on the drama dataset (denoted as Drama222Pseudo labels obtained from Gemini-3 are treated as ground truth) and the real-world recording dataset TaigiSpeech (denoted as Taigi).

## Citation

```bibtex
@misc{chang2026taigispeech,
  title={TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild},
  author={Kai-Wei Chang et al.},
  year={2026},
  note={arXiv:2603.21478}
}
```

- arXiv: 2603.21478