Minisuperb Eval

---
name: minisuperb-eval
description: Evaluates self-supervised speech models by measuring computational efficiency (forward MACs) and downstream task performance using a lightweight, offline feature extraction protocol. It probes the trade-off between model complexity and representation quality across speech tasks while enabling rapid early-stage model screening. Use when the user wants to benchmark on MiniSUPERB, or asks about evaluating this task. Reports forward MACs.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.19011
  bibtex_key: wang2023minisuperb
  confidence: medium
---

# minisuperb-eval

> MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models — Wang et al. (2023) (arXiv:2305.19011, 2023)

## What this evaluates

Evaluates self-supervised speech models by measuring computational efficiency (forward MACs) and downstream task performance using a lightweight, offline feature extraction protocol. It probes the trade-off between model complexity and representation quality across speech tasks while enabling rapid early-stage model screening.

## Datasets

- **MiniSUPERB** — total ?; splits: test (32); repo https://github.com/Comet0322/MiniSUPERB

## Metrics

- `forward MACs` **(primary)** — range: other
  - Estimates the total number of multiply-accumulate operations (MACs) for the forward pass of both the upstream and downstream models.

## Input / output format

**Input**: Speech utterances (32 samples from LibriSpeech test-clean) fed into a frozen upstream self-supervised speech model for offline representation extraction.

**Output**: Layer-wise hidden representations from the upstream model, processed through layer normalization and a weighted sum with trainable weights, then passed to the downstream model. Forward MAC count estimated via Microsoft DeepSpeed.

## Scoring recipe

```python
# Estimate forward MACs for upstream and downstream models
upstream_macs = deepspeed.estimate_macs(upstream_model, input_utterances, batch_size=1)
downstream_macs = deepspeed.estimate_macs(downstream_model, extracted_representations)
total_macs = upstream_macs + downstream_macs
return total_macs
```

## Common pitfalls

- Using batch sizes > 1 during offline feature extraction can introduce padding and normalization artifacts that distort MAC estimates.
- Separating upstream and downstream computation improves training efficiency but may not reflect real-world inference latency where both run jointly.
- Evaluating only on 32 utterances provides a proxy for computational cost but may not capture full dataset performance variance.

## Evidence (verbatim from paper)

> To evaluate computational costs, we follow the setting in SUPERB Challenge and rely on Microsoft DeepSpeed333<https://github.com/microsoft/DeepSpeed> to estimate the forward MACs of both the upstream and downstream models. Following SUPERB Challenge, we perform this evaluation over the same 32 utterances sampled from the LibriSpeech test-clean dataset as inputs.

## Citation

```bibtex
@misc{wang2023minisuperb,
  title={MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2305.19011}
}
```

- arXiv: 2305.19011