Sytts Commands Eval

---
name: sytts-commands-eval
description: Evaluates the quality and effectiveness of a synthetic multilingual voice command dataset for on-device keyword spotting. It probes whether TTS-synthesized audio can support high-accuracy classification across different model complexities and languages (English and Chinese). Use when the user wants to benchmark on SYNTTS-COMMANDS, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.07821
  bibtex_key: gan2025syttscommands
  confidence: high
---

# sytts-commands-eval

> SynTTS-Commands: A Public Dataset for On-Device KWS via TTS-Synthesized Multilingual Speech — Gan et al. (2025) (arXiv:2511.07821, 2025)

## What this evaluates

Evaluates the quality and effectiveness of a synthetic multilingual voice command dataset for on-device keyword spotting. It probes whether TTS-synthesized audio can support high-accuracy classification across different model complexities and languages (English and Chinese).

## Datasets

- **SYNTTS-COMMANDS** — total ?; splits: train (-1), test (-1)

## Metrics

- `classification accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted command labels out of the total number of test instances. Calculated as correct predictions divided by total predictions for multi-class classification.
- `cross-entropy loss` — range: [0, ∞)
  - Standard categorical cross-entropy loss measuring the divergence between predicted probability distributions and true labels. Lower values indicate better predictive confidence and calibration.

## Input / output format

**Input**: Audio recordings of voice commands in English or Chinese, converted into acoustic features for classification models.

**Output**: Discrete command class labels (one per audio instance).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    accuracy = (predictions == gold_labels).mean()
    loss = categorical_cross_entropy(predictions, gold_labels)
    return {'accuracy': accuracy, 'loss': loss}
```

## Common pitfalls

- Parameter counts vary between English and Chinese deployments due to different output class sizes, not architecture changes, which can mislead size comparisons.
- Lightweight models show significant accuracy drops on Chinese due to tonal/phonetic complexity, which may be mistaken for dataset quality issues rather than model capacity limits.
- Higher accuracy does not always correlate with lower loss; models like EfficientNet-B0 show better calibration despite slightly higher loss.

## Evidence (verbatim from paper)

> Performance was measured using classification accuracy and cross-entropy loss, enabling a holistic assessment of both predictive power and calibration across languages and model sizes.

## Citation

```bibtex
@misc{gan2025syttscommands,
  title={SynTTS-Commands: A Public Dataset for On-Device KWS via TTS-Synthesized Multilingual Speech},
  author={Gan et al. (2025)},
  year={2025},
  note={arXiv:2511.07821}
}
```

- arXiv: 2511.07821