Ml Superb Eval

---
name: ml-superb-eval
description: Evaluates multilingual speech processing capabilities across 143 languages on ASR, Language Identification, and joint tasks under normal and few-shot settings. It probes cross-lingual transfer and low-resource adaptation of SSL models. Use when the user wants to benchmark on ML-SUPERB, or asks about evaluating this task. Reports ML-SUPERB score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.02720
  bibtex_key: shi2023multiresolutionhubert
  confidence: high
---

# ml-superb-eval

> Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction — Shi et al. (2023) (arXiv:2310.02720, 2023)

## What this evaluates

Evaluates multilingual speech processing capabilities across 143 languages on ASR, Language Identification, and joint tasks under normal and few-shot settings. It probes cross-lingual transfer and low-resource adaptation of SSL models.

## Datasets

- **ML-SUPERB** — total ?; splits: 10-minute (-1), 1-hour (-1); repo https://github.com/espnet/espnet/tree/master/egs2/ml_superb/asr1

## Metrics

- `ML-SUPERB score` **(primary)** — range: other
  - Composite score defined by Shi et al. (2023a) aggregating performance across Monolingual ASR, Multilingual ASR, LID, and joint tasks. Task metrics include CER/PER and Accuracy.

## Input / output format

**Input**: Frozen SSL representations fed into multilingual downstream architectures.

**Output**: Task-specific predictions (e.g., transcripts, language labels).

## Scoring recipe

```python
def compute_ml_superb_score(task_metrics):
    # Compute CER/PER for ASR tasks and Accuracy for LID
    # Aggregate into composite ML-SUPERB score per benchmark set (10-min/1-hour)
    ml_superb_score = aggregate_multilingual_metrics(task_metrics)
    return ml_superb_score
```

## Common pitfalls

- Few-shot settings use very limited labeled data per language, making results highly sensitive to the downstream training recipe.
- The composite score masks per-language performance variations across the 143 languages.

## Evidence (verbatim from paper)

> We evaluate the performance of our proposed multilingual speech processing method using the ML-SUPERB benchmark... The ML-SUPERB benchmark comprises two sets of general benchmarks—specifically, a 10-minute set and a 1-hour set—across four tasks... we calculate a composite ML-SUPERB score as defined by Shi et al. (2023a)...

## Citation

```bibtex
@misc{shi2023multiresolutionhubert,
  title={Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction},
  author={Shi et al. (2023)},
  year={2023},
  note={arXiv:2310.02720}
}
```

- arXiv: 2310.02720