Frill Noss Esc50 Eval

---
name: frill-noss-esc50-eval
description: Evaluates the quality of lightweight, non-semantic speech embeddings by training simple downstream classifiers on averaged embedding features to perform audio classification tasks. It also measures inference latency on a mobile device to assess real-time suitability for on-device deployment. Use when the user wants to benchmark on NOSS benchmark, ESC-50 (human sounds subset), Mask speech dataset, or asks about evaluating this task. Reports test accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.04609
  bibtex_key: peplinski2020frill
  confidence: high
---

# frill-noss-esc50-eval

> FRILL: A Non-Semantic Speech Embedding for Mobile Devices — Peplinski et al. (2020) (arXiv:2011.04609, 2020)

## What this evaluates

Evaluates the quality of lightweight, non-semantic speech embeddings by training simple downstream classifiers on averaged embedding features to perform audio classification tasks. It also measures inference latency on a mobile device to assess real-time suitability for on-device deployment.

## Datasets

- **NOSS benchmark** — total ?; splits: train (-1), test (-1)
- **ESC-50 (human sounds subset)** — total ?; splits: train (-1), test (-1)
- **Mask speech dataset** — total 37000; splits: train (-1), test (-1)

## Metrics

- `test accuracy` **(primary)** — range: [0, 1]
  - Accuracy = (number of correct predictions) / (total number of predictions). Reported as the best score across logistic regression, random forest, and LDA classifiers, with or without L2 speaker normalization.
- `unweighted average recall` — range: [0, 1]
  - Mean of recall scores computed per class, without weighting by class frequency. Used as an indicator for the mask speech task.

## Input / output format

**Input**: Log-magnitude Mel spectrograms (64 bins, 25ms window, 10ms stride) derived from audio clips, or averaged per-utterance embedding vectors for downstream classification.

**Output**: Classification labels for downstream tasks; single embedding vector per utterance for latency/size benchmarking.

## Scoring recipe

```python
def evaluate(model, spectrograms, labels, dataset):
    embeddings = model(spectrograms)
    features = mean(embeddings, dim=time)
    if dataset in [SpeechCommands, CREMA-D, SAVEE]:
        features = l2_normalize(features, dim=speaker)
    best_acc = 0
    for clf in [LogisticRegression, RandomForest, LDA]:
        clf.fit(train_features, train_labels)
        preds = clf.predict(test_features)
        acc = accuracy_score(test_labels, preds)
        best_acc = max(best_acc, acc)
    return best_acc
```

## Common pitfalls

- Embeddings are averaged over time per utterance, which discards temporal dynamics and may hurt performance on tasks requiring fine-grained temporal patterns.
- Speaker normalization (L2) is only applied to specific datasets (SpeechCommands, CREMA-D, SAVEE), not consistently across all benchmarks.
- Test labels for the mask speech task were unavailable at publication, so performance is reported as an indicator rather than a strict benchmark score.

## Evidence (verbatim from paper)

> We report the best test accuracy across combinations of downstream classifiers and normalization techniques. ... We use the first four published folds of ESC-50 for training, and the fifth for testing.

## Citation

```bibtex
@misc{peplinski2020frill,
  title={FRILL: A Non-Semantic Speech Embedding for Mobile Devices},
  author={Peplinski et al. (2020)},
  year={2020},
  note={arXiv:2011.04609}
}
```

- arXiv: 2011.04609