Nlu Service Eval

---
name: nlu-service-eval
description: Evaluates commercial and open-source NLU platforms on intent classification and named entity recognition across multiple dialogue domains, highlighting limitations in multi-intent support and contextual modeling. Use when the user wants to benchmark on NLU Evaluation Dataset, or asks about evaluating this task. Reports Intent classification accuracy, Entity recognition precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1903.05566
  bibtex_key: liu2019nlu
  confidence: high
---

# nlu-service-eval

> Benchmarking Natural Language Understanding Services for building Conversational Agents — Liu et al. (2019) (arXiv:1903.05566, 2019)

## What this evaluates

Evaluates commercial and open-source NLU platforms on intent classification and named entity recognition across multiple dialogue domains, highlighting limitations in multi-intent support and contextual modeling.

## Datasets

- **NLU Evaluation Dataset** — total 11036; splits: train (-1), test (-1); repo https://github.com/xliuhw/NLU-Evaluation-Data

## Metrics

- `Intent classification accuracy, Entity recognition precision` **(primary)** — range: percent
  - Standard classification metrics: accuracy for intent classification, and precision/recall/F1 for entity recognition. The paper specifically highlights precision differences for entity recognition.

## Input / output format

**Input**: Raw user utterance text.

**Output**: Predicted intent label(s) and extracted entity spans with types for each utterance.

## Scoring recipe

```python
# 10-fold cross-validation (90% train, 10% test per fold)
for fold in range(10):
    train_set = dataset[fold * 0.9 : (fold + 1) * 0.9]
    test_set = dataset[(fold + 1) * 0.9 : (fold + 1)]
    predictions = call_nlu_api(test_set)
    intent_acc = sum(p.intent == g.intent for p, g in zip(predictions, test_set)) / len(test_set)
    entity_prec = compute_entity_precision(predictions, test_set)
    # Aggregate metrics across folds
```

## Common pitfalls

- Dataset is inherently unbalanced across intents and entities (e.g., some intents have as few as 77 instances), which may skew per-class metrics.
- API versioning and configuration differences across platforms (e.g., Dialogflow V1.0 vs V2.0, Watson version matching) can significantly impact results.
- Platforms lack support for multiple intents per utterance and dialogue context, limiting joint modeling evaluation.

## Evidence (verbatim from paper)

> For the evaluation experiments we report below, we performed 10 fold cross-validation with 90% of the subcorpus for training and 10% for testing in each fold. ... Watson leads in Intent classification but lags in Entity Recognition due to low precision

## Citation

```bibtex
@misc{liu2019nlu,
  title={Benchmarking Natural Language Understanding Services for building Conversational Agents},
  author={Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05566}
}
```

- arXiv: 1903.05566