Hear Eval

---
name: hear-eval
description: Evaluates the zero-shot generalization and transferability of pre-trained audio representations across 19 diverse downstream tasks spanning speech, environmental sounds, and music. The benchmark requires models to perform without fine-tuning, emphasizing robustness and cross-domain adaptability. Use when the user wants to benchmark on FSD50K, ESC-50, GTZAN, Vocal Imitations, LibriCount, CREMA-D, VoxLingua107, Speech Commands, DCASE 2016 Task 2, Gunshot Triangulation, Beijing Opera, Mridingham Stroke and Tonic, NSynth, Maestro, or asks about evaluating this task. Reports normalized score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.03022
  bibtex_key: turian2022hear
  confidence: high
---

# hear-eval

> HEAR: Holistic Evaluation of Audio Representations — Turian et al. (2022) (arXiv:2203.03022, 2022)

## What this evaluates

Evaluates the zero-shot generalization and transferability of pre-trained audio representations across 19 diverse downstream tasks spanning speech, environmental sounds, and music. The benchmark requires models to perform without fine-tuning, emphasizing robustness and cross-domain adaptability.

## Datasets

- **FSD50K** — total ?; splits: test (-1)
- **ESC-50** — total ?; splits: test (-1)
- **GTZAN** — total ?; splits: test (-1)
- **Vocal Imitations** — total ?; splits: test (-1)
- **LibriCount** — total ?; splits: test (-1)
- **CREMA-D** — total ?; splits: test (-1)
- **VoxLingua107** — total ?; splits: test (-1)
- **Speech Commands** — total ?; splits: test (-1)
- **DCASE 2016 Task 2** — total ?; splits: test (-1)
- **Gunshot Triangulation** — total ?; splits: test (-1)
- **Beijing Opera** — total ?; splits: test (-1)
- **Mridingham Stroke and Tonic** — total ?; splits: test (-1)
- **NSynth** — total ?; splits: test (-1)
- **Maestro** — total ?; splits: test (-1)

## Metrics

- `normalized score` **(primary)** — range: [-1, 1]
  - Per-task scores are standardized to zero mean and unit variance, then Winsorized (clamped) to [-1, +1] to limit extreme outliers. The final score is the arithmetic mean across all tasks.
- `mean top-1 accuracy` — range: [0, 1]
  - Standard top-1 classification accuracy averaged across tasks.
- `mean average precision (mAP)` — range: [0, 1]
  - Mean average precision for multi-label or ranking tasks.

## Input / output format

**Input**: Raw audio clips or segments corresponding to the specific downstream task (e.g., classification, onset detection, pitch estimation, counting).

**Output**: Task-specific predictions: class labels, counts, pitch values, or temporal onsets, depending on the task.

## Scoring recipe

```python
def compute_hear_score(task_scores, task_means, task_stds):
    standardized = {}
    for task, score in task_scores.items():
        standardized[task] = (score - task_means[task]) / task_stds[task]
    winsorized = {t: max(-1.0, min(1.0, s)) for t, s in standardized.items()}
    return sum(winsorized.values()) / len(winsorized)
```

## Common pitfalls

- Evaluating with fine-tuned models instead of the required zero-shot setting.
- Skipping the Winsorization step when aggregating scores, which allows extreme outliers on single tasks to skew inter-task comparisons.
- Relying solely on a single aggregate score, which strips out important task-specific performance nuances highlighted in the full score table.

## Evidence (verbatim from paper)

> Zhai et al. (2019) compare a variety of aggregation techniques for evaluating cross-task model performance, and find that they are all highly correlated, settling upon simple mean top-1. ... For these summary figures, we normalize each model/task score. Normalized scores allow us to compare models and tasks against each other, under the assumption each task is equally weighted. The normalization procedure is as follows: 1) For each task, we standardize the scores to zero mean and unit variance. Unlike transforming tasks to ranks, we assume that the scale of intra-task scores is important. 2) The standardized scores are Winsorized (clamped) to have variance within $[-1, +1]$ . By limiting the importance of extremely high or low scores on a single task, this approach allows for better inter-task comparison.

## Citation

```bibtex
@misc{turian2022hear,
  title={HEAR: Holistic Evaluation of Audio Representations},
  author={Turian et al. (2022)},
  year={2022},
  note={arXiv:2203.03022}
}
```

- arXiv: 2203.03022