Intent Detection Eval

---
name: intent-detection-eval
description: This benchmark evaluates the ability of pretrained sentence encoders and classifiers to correctly identify user intents from conversational utterances. It specifically probes few-shot generalization by testing models on severely limited training data (10 or 30 examples per intent) while maintaining a standard full test set. Use when the user wants to benchmark on BANKING77, CLINC150, HWU64, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.04807
  bibtex_key: casanueva2020efficient
  confidence: high
---

# intent-detection-eval

> Efficient Intent Detection with Dual Sentence Encoders — Casanueva et al. (2020) (arXiv:2003.04807, 2020)

## What this evaluates

This benchmark evaluates the ability of pretrained sentence encoders and classifiers to correctly identify user intents from conversational utterances. It specifically probes few-shot generalization by testing models on severely limited training data (10 or 30 examples per intent) while maintaining a standard full test set.

## Datasets

- **BANKING77** — total ?; splits: train (-1), test (-1)
- **CLINC150** — total ?; splits: train (-1), test (-1)
- **HWU64** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted intent labels out of the total number of test instances. Reported as a percentage (×100%).

## Input / output format

**Input**: A single user utterance (text string), encoded via a pretrained sentence encoder.

**Output**: A predicted intent class label corresponding to one of the predefined intent categories.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Few-shot training sets contain only 10 or 30 examples *per intent*, not 10 or 30 total across all intents.
- The test set is never subsampled; it remains the full standard test set for every experimental run.
- BERT-FIXED uses mean-pooled sequence output for encoding, whereas BERT-TUNED fine-tunes the full model and uses the [CLS] token output.

## Evidence (verbatim from paper)

> We report accuracy as the main evaluation measure for all experimental runs. We experiment with setups where only 10 or 30 examples are available for each intent, while we use the same standard test sets for each experimental run.

## Citation

```bibtex
@misc{casanueva2020efficient,
  title={Efficient Intent Detection with Dual Sentence Encoders},
  author={Casanueva et al. (2020)},
  year={2020},
  note={arXiv:2003.04807}
}
```

- arXiv: 2003.04807