Dynamic Superb Phase2 Eval

---
name: dynamic-superb-phase2-eval
description: Evaluates instruction-based universal speech and audio models across 180 tasks spanning speech, music, and environmental audio. It probes capabilities like automatic speech recognition, emotion recognition, speaker verification, and audio classification using a unified instruction-following framework. Use when the user wants to benchmark on Dynamic-SUPERB Phase-2, or asks about evaluating this task. Reports relative_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.05361
  bibtex_key: [REDACTED]
  confidence: high
---

# dynamic-superb-phase2-eval

> Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks — Huang et al. (2024) (arXiv:2411.05361, 2024)

## What this evaluates

Evaluates instruction-based universal speech and audio models across 180 tasks spanning speech, music, and environmental audio. It probes capabilities like automatic speech recognition, emotion recognition, speaker verification, and audio classification using a unified instruction-following framework.

## Datasets

- **Dynamic-SUPERB Phase-2** — total ?; splits: test (-1); repo https://github.com/dynamic-superb/dynamic-superb

## Metrics

- `relative_score` **(primary)** — range: percent
  - Domain-level score averaging task-level relative improvements against a Whisper+LLaMA baseline. Task metrics are first scaled by the N/A rate: higher-is-better metrics are multiplied by (1 - N/A_rate), lower-is-better metrics are divided by (1 - N/A_rate). Relative improvement is then averaged per domain.
- `WER` — range: percent
  - Word Error Rate, standard ASR metric counting substitutions, deletions, and insertions normalized by reference words.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted class labels out of total instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for slot-filling and other classification tasks.
- `PER` — range: percent
  - Phoneme Error Rate, analogous to WER but at the phoneme level.
- `CER` — range: percent
  - Character Error Rate for slot value generation tasks.
- `DER` — range: percent
  - Diarization Error Rate measuring speaker turn confusion and misclassification.

## Input / output format

**Input**: Raw audio waveform paired with a natural language instruction/prompt specifying the task (e.g., transcribe, classify, detect emotion, generate text/audio).

**Output**: Task-dependent outputs: text transcriptions, class labels, numerical values, or generated audio/text sequences. Models must strictly follow the instruction format to avoid high N/A rates.

## Scoring recipe

```python
def compute_domain_relative_score(tasks):
    domain_scores = []
    for task in tasks:
        raw_metric = evaluate_task(task.model_output, task.gold)
        baseline_metric = evaluate_task(task.baseline_output, task.gold)
        na_rate = count_na(task.model_output) / len(task.gold)
        
        if task.metric_direction == 'higher_is_better':
            scaled = raw_metric * (1 - na_rate)
            baseline_scaled = baseline_metric * (1 - na_rate)
        else:
            scaled = raw_metric / (1 - na_rate)
            baseline_scaled = baseline_metric / (1 - na_rate)
            
        relative_improvement = (scaled - baseline_scaled) / baseline_scaled
        domain_scores.append(relative_improvement)
    return sum(domain_scores) / len(domain_scores)
```

## Common pitfalls

- Directly comparing Dynamic-SUPERB results to original SUPERB/HEAR benchmarks without accounting for the instruction-following adaptation and N/A rate scaling.
- Ignoring the N/A rate penalty for regression/generation tasks, which heavily penalizes models that fail to follow formatting instructions.
- Domain-level scores can be distorted by outliers with unbounded metrics (e.g., phoneme counting tasks with massive MSE).

## Evidence (verbatim from paper)

> For each task, we calculated the relative improvement of each model compared to the cascaded system baseline (Whisper + LLaMA) and then obtained the domain-level scores by averaging all its improvements across the tasks within each domain. For regression tasks, we introduced the N/A rate to measure how well a model meets the task requirements. The original task metric was only computed on instances that followed the task format (after post-processing by the LLM). To account for whether a model can follow instructions, we incorporated the N/A rate into the reported scores. Accordingly, we calculated the improvement based on scaled values. For metrics where a higher value indicates better performance (such as the F1 score), we multiplied the metric value by (1 – N/A rate). Conversely, for metrics where a lower value indicates better performance (such as word error rate), we divided the metric value by (1 – N/A rate).

## Citation

```bibtex
@misc{huang2024dynamicsuperb,
  title={Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2411.05361}
}
```

- arXiv: 2411.05361