Toyadmos Eval

---
name: toyadmos-eval
description: Evaluates unsupervised anomalous sound detection systems on miniature machine operating sounds. It probes the ability of models to learn normal acoustic patterns and identify deviations caused by mechanical faults or environmental variations. Use when the user wants to benchmark on ToyADMOS, or asks about evaluating this task. Reports AUC-ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.03299
  bibtex_key: [REDACTED]
  confidence: high
---

# toyadmos-eval

> ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection — Koizumi et al. (2019) (arXiv:1908.03299, 2019)

## What this evaluates

Evaluates unsupervised anomalous sound detection systems on miniature machine operating sounds. It probes the ability of models to learn normal acoustic patterns and identify deviations caused by mechanical faults or environmental variations.

## Datasets

- **ToyADMOS** — total ?; splits: train (1000), test (-1); repo https://github.com/YumaKoizumi/ToyADMOS-dataset

## Metrics

- `AUC-ROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve computed over file-level binary predictions derived from frame-level anomaly scores.

## Input / output format

**Input**: 16 kHz audio wav-files, converted to 64-dimensional log-mel-amplitude spectra with 10 preceding and 10 succeeding time-frames concatenated as context.

**Output**: Binary file-level label (anomalous/normal) determined by whether the maximum frame-level reconstruction error exceeds a fixed threshold.

## Scoring recipe

```python
# For each test wav-file:
frame_scores = [reconstruction_error(frame) for frame in wav_file_frames]
file_score = max(frame_scores)
prediction = 1 if file_score > threshold else 0
# Compute AUC-ROC over all file predictions vs ground truth labels
```

## Common pitfalls

- The toy-train sub-dataset mixes all 4 microphone channels into a single channel, unlike the others.
- File-level anomaly is declared if *any* single frame exceeds the threshold, not an average or majority vote.
- Environmental noise is mixed at a specific SNR (+10 dB) during preprocessing, which significantly affects baseline performance.

## Evidence (verbatim from paper)

> We calculated the anomaly scores on each time frame of all test wav-files. If the anomaly score exceeded the threshold even for one frame, the wav-file was determined to be anomalous. This system gave the area under the receiver operating characteristic curves of 0.874, 0.981, and 0.843 for the toy-car, toy-conveyor, and toy-train sub-datasets, respectively.

## Citation

```bibtex
@misc{koizumi2019toyadmos,
  title={ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection},
  author={Koizumi et al. (2019)},
  year={2019},
  note={arXiv:1908.03299}
}
```

- arXiv: 1908.03299