Speechinstructbench Eval

---
name: speechinstructbench-eval
description: Evaluates speech instruction-following capabilities across closed-ended, open-ended, and adjustment tasks under varying acoustic conditions (background noise, accents, disfluencies) in English and Chinese. Use when the user wants to benchmark on SpeechInstructBench, or asks about evaluating this task. Reports instruction-level accuracy (I).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.02769
  bibtex_key: wang2025inserter
  confidence: high
---

# speechinstructbench-eval

> InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training — Wang et al. (2025) (arXiv:2503.02769, 2025)

## What this evaluates

Evaluates speech instruction-following capabilities across closed-ended, open-ended, and adjustment tasks under varying acoustic conditions (background noise, accents, disfluencies) in English and Chinese.

## Datasets

- **SpeechInstructBench** — total ?; splits: test (-1)

## Metrics

- `prompt-level accuracy (P)` — range: percent
  - Computed by averaging loose and strict accuracies for closed-ended questions.
- `instruction-level accuracy (I)` **(primary)** — range: percent
  - Computed by averaging loose and strict accuracies for closed-ended questions.
- `Instruction Adherence Rate (IAR)` — range: percent
  - Measures the rate of correct instruction adherence for adjustment tasks; higher is better.
- `Error Correction Rate (ECR)` — range: percent
  - Measures the error correction rate for adjustment tasks; lower is better.

## Input / output format

**Input**: 16kHz resampled speech segments converted to 128-channel mel-spectrograms (25ms window, 10ms hop, stride-2 pooling), provided alongside text prompts or instructions.

**Output**: Text response predicting the continuation or answering the instruction.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    loose_acc = match_loose(predictions, gold)
    strict_acc = match_strict(predictions, gold)
    p_acc = (loose_acc + strict_acc) / 2
    i_acc = (loose_acc + strict_acc) / 2
    iar = count_adherence(predictions, gold) / len(gold)
    ecr = count_errors(predictions, gold) / len(gold)
    return {'P': p_acc, 'I': i_acc, 'IAR': iar, 'ECR': ecr}
```

## Common pitfalls

- P and I accuracies are averaged from loose and strict versions, not reported separately.
- ECR is a lower-is-better metric (↓) while IAR is higher-is-better (↑).
- Chinese benchmark excludes models without Chinese response capabilities (e.g., Mini-Omni variants).

## Evidence (verbatim from paper)

> In SpeechInstructBench, evaluation metrics encompass prompt-level (P) and instruction-level (I) accuracy for both closed-ended and open-ended tasks. For closed-ended questions, P and I metrics are computed by averaging loose and strict accuracies, while adjustment task is assessed using Instruction Adherence Rate (IAR) and Error Correction Rate (ECR).

## Citation

```bibtex
@misc{wang2025inserter,
  title={InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.02769}
}
```

- arXiv: 2503.02769