Slu Eval

---
name: slu-eval
description: Evaluates spoken language understanding capabilities across multiple tasks including intent classification, slot filling, emotion recognition, and dialogue act classification. It probes a model's ability to map raw audio inputs to semantic labels, test robustness to noise and low-resource settings, and assess the utility of pretrained ASR/NLU feature extractors in end-to-end speech processing pipelines. Use when the user wants to benchmark on FSC (Fluent Speech Commands), Snips, SLURP, IEMOCAP, Switchboard (NXT-format), Grabo, CAT-SLU MAP, Google Speech Commands, HarperValleyBank, or asks about evaluating this task. Reports Intent Classification Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.14706
  bibtex_key: arora2021espnetslu
  confidence: high
---

# slu-eval

> ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet — Siddhant Arora et al. (2021) (arXiv:2111.14706, 2021)

## What this evaluates

Evaluates spoken language understanding capabilities across multiple tasks including intent classification, slot filling, emotion recognition, and dialogue act classification. It probes a model's ability to map raw audio inputs to semantic labels, test robustness to noise and low-resource settings, and assess the utility of pretrained ASR/NLU feature extractors in end-to-end speech processing pipelines.

## Datasets

- **FSC (Fluent Speech Commands)** — total ?; splits: test (-1)
- **Snips** — total ?; splits: train (-1), test (-1)
- **SLURP** — total ?; splits: train (-1), test (-1)
- **IEMOCAP** — total ?; splits: train (-1), test (-1)
- **Switchboard (NXT-format)** — total ?; splits: test (-1)
- **Grabo** — total ?; splits: test (-1)
- **CAT-SLU MAP** — total ?; splits: test (-1)
- **Google Speech Commands** — total ?; splits: test (-1)
- **HarperValleyBank** — total ?; splits: test (-1)

## Metrics

- `Intent Classification Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted intent labels out of total test instances. F1 is also reported in some tables as a macro-averaged harmonic mean of precision and recall.
- `Slot Filling F1` — range: [0, 1]
  - Macro-averaged F1 score for entity and slot label prediction.
- `Emotion Recognition Accuracy` — range: percent
  - Percentage of correctly predicted emotion classes (neutral, happy, sad, angry) out of total instances.
- `Dialogue Act Accuracy` — range: percent
  - Percentage of correctly predicted dialogue act classes out of total instances.

## Input / output format

**Input**: Raw audio waveform files, optionally concatenated with transcripts or acoustic signals from preceding utterances for context. Noisy variants include added real-world noise.

**Output**: Single token or word representing the predicted intent, emotion, or dialogue act class. For slot filling: intent token followed by entity label and lexical filler tokens, separated by special separator tokens.

## Scoring recipe

```python
def compute_metric(predictions, gold_labels, metric_type='accuracy'):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels) * 100
    elif metric_type == 'f1':
        from sklearn.metrics import f1_score
        return f1_score(gold_labels, predictions, average='macro') * 100
```

## Common pitfalls

- Splits are not standardized across datasets; authors use original splits but adopt alternative random splits for Snips, complicating direct comparison.
- Pretrained ASR/NLU feature extractors do not consistently improve performance; on SLURP, raw FBANK features outperformed HuBERT/Wav2vec2.
- Vocabulary mismatch between ASR transcripts and intent/slot labels can degrade performance if not mitigated (e.g., via BPE).

## Evidence (verbatim from paper)

> Table 5. Intent Classification accuracy on the SLURP Dataset [4] where we perform comparison between different pretrained ASR and NLU systems as feature extractors.

## Citation

```bibtex
@misc{arora2021espnetslu,
  title={ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet},
  author={Siddhant Arora et al. (2021)},
  year={2021},
  note={arXiv:2111.14706}
}
```

- arXiv: 2111.14706