Mimii Dg Eval

---
name: mimii-dg-eval
description: Evaluates domain generalization capabilities for anomalous sound detection by measuring how well models trained on a source domain of industrial machine sounds generalize to target domains with shifted operational parameters or background noise. Use when the user wants to benchmark on MIMII DG, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.13879
  bibtex_key: dohi2022mimiidg
  confidence: high
---

# mimii-dg-eval

> MIMII DG: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization Task — Dohi et al. (2022) (arXiv:2205.13879, 2022)

## What this evaluates

Evaluates domain generalization capabilities for anomalous sound detection by measuring how well models trained on a source domain of industrial machine sounds generalize to target domains with shifted operational parameters or background noise.

## Datasets

- **MIMII DG** — total ?; splits: source (-1), target (-1)

## Metrics

- `AUC` **(primary)** — range: percent
  - Area under the receiver operating characteristic curve. Calculated per domain, section, and machine type as: AUC = (1/(N_d^- * N_n^+)) * sum_{i=1}^{N_d^-} sum_{j=1}^{N_n^+} H(A_theta(x_j^+) - A_theta(x_i^-)), where H is the Heaviside step function, A_theta is the anomaly score, N_d^- is the number of normal test clips in domain d, and N_n^+ is the number of anomalous test clips in section n.

## Input / output format

**Input**: Audio clips from industrial machines converted to log-Mel spectrograms (e.g., 640-dimensional vectors for autoencoder, concatenated frames for MobileNetV2).

**Output**: Anomaly score per audio clip.

## Scoring recipe

```python
def compute_auc(anomaly_scores_normal, anomaly_scores_anomalous):
    n_normal = len(anomaly_scores_normal)
    n_anomalous = len(anomaly_scores_anomalous)
    if n_normal == 0 or n_anomalous == 0:
        return 0.0
    correct_predictions = 0
    for score_normal in anomaly_scores_normal:
        for score_anomalous in anomaly_scores_anomalous:
            if score_anomalous > score_normal:
                correct_predictions += 1
    return correct_predictions / (n_normal * n_anomalous)
```

## Common pitfalls

- The dataset evaluates domain generalization, so models must be trained only on source domain data and tested on both source and target domains without adaptation.
- AUC is computed separately for each domain, section, and machine type before averaging; reporting a single global AUC across all domains obscures the domain shift effect.
- Anomaly scores are threshold-independent for AUC, but the paper notes the detector should ideally work with the same threshold across domains for practical deployment.

## Evidence (verbatim from paper)

> We used the area under the receiver operating characteristic curve (AUC) for evaluation. Because the domain generalization task requires detecting anomalies even when the occurance of domain shifts can be difficult to detect, the anomaly detector is expected to work with the same threshold regardless of the domain. Therefore, we calculated the AUC using both the source and target domain data. Also, to evaluate the anomaly detection performance for each domain, the AUC was computed for each domain. The AUC for each domain, section, and machine type was calculated as

## Citation

```bibtex
@misc{dohi2022mimiidg,
  title={MIMII DG: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization Task},
  author={Dohi et al. (2022)},
  year={2022},
  note={arXiv:2205.13879}
}
```

- arXiv: 2205.13879