Ccl Slu Eval

---
name: ccl-slu-eval
description: Evaluates intent classification robustness under noisy ASR conditions by measuring how well a model aligns noisy transcripts with clean references and preserves semantic consistency. Use when the user wants to benchmark on SLURP, Timers, FSC, SNIPS, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.15097
  bibtex_key: kim2024contrastive
  confidence: medium
---

# ccl-slu-eval

> Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding — Kim et al. (2024) (arXiv:2405.15097, 2024)

## What this evaluates

Evaluates intent classification robustness under noisy ASR conditions by measuring how well a model aligns noisy transcripts with clean references and preserves semantic consistency.

## Datasets

- **SLURP** — total ?; splits: test (-1)
- **Timers** — total ?; splits: test (-1)
- **FSC** — total ?; splits: test (-1)
- **SNIPS** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted intent labels out of total instances.
- `Macro F1-score` — range: [0, 1]
  - Unweighted mean of F1 scores computed per intent class, then averaged across all classes.

## Input / output format

**Input**: Noisy or clean ASR transcripts (text strings), optionally tokenized, provided as input to the encoder.

**Output**: Predicted intent class label.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = sorted(set(golds))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    macro_f1 = sum(f1s) / len(f1s)
    return acc, macro_f1
```

## Common pitfalls

- ASR error rates (WER) vary significantly across datasets; SLURP and Timers have higher WER due to diverse accents, while FSC has the lowest.
- Performance is highly sensitive to noise levels (e.g., Noisy0.24 vs Noisy0.56), and models must explicitly handle insertion, deletion, and substitution errors rather than just learning clean mappings.

## Evidence (verbatim from paper)

> Table 3: Results for macro F1-score and accuracy performances on SLURP dataset for our model trained with CCL method. We compare the CCL method and other baselines for clean and noisy transcripts.

## Citation

```bibtex
@misc{kim2024contrastive,
  title={Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2405.15097}
}
```

- arXiv: 2405.15097