Mixatis Mixsnips Eval

---
name: mixatis-mixsnips-eval
description: Evaluates joint intent detection and slot filling on mixed-domain conversational datasets. It probes a model's ability to simultaneously predict multiple intents per utterance and extract corresponding slot entities, measuring both token-level and sentence-level alignment. Use when the user wants to benchmark on MixATIS, MixSNIPS, or asks about evaluating this task. Reports Overall Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.10375
  bibtex_key: xing2022coguidingnet
  confidence: high
---

# mixatis-mixsnips-eval

> Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs — Xing et al. (2022) (arXiv:2210.10375, 2022)

## What this evaluates

Evaluates joint intent detection and slot filling on mixed-domain conversational datasets. It probes a model's ability to simultaneously predict multiple intents per utterance and extract corresponding slot entities, measuring both token-level and sentence-level alignment.

## Datasets

- **MixATIS** — total 14746; splits: train (13162), val (756), test (828)
- **MixSNIPS** — total 44173; splits: train (39776), val (2198), test (2199)

## Metrics

- `Intent Accuracy` — range: percent
  - Percentage of utterances where the predicted set of intents exactly matches the ground truth set.
- `Slot F1` — range: percent
  - Standard token-level F1 score for slot labeling, computed over all tokens across the dataset.
- `Overall Accuracy` **(primary)** — range: percent
  - Ratio of sentences where both the predicted intent set and all predicted slot labels exactly match the ground truth.

## Input / output format

**Input**: Raw text utterance

**Output**: A set of predicted intent labels and a sequence of token-aligned slot labels (e.g., BIO format)

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    intent_correct = (set(predictions.intents) == set(gold.intents))
    slot_correct = (predictions.slots == gold.slots)
    overall_acc = 1.0 if (intent_correct and slot_correct) else 0.0
    return overall_acc
# Note: Slot F1 and Intent Acc are computed independently across the dataset using standard token-level and set-matching definitions.
```

## Common pitfalls

- Overall accuracy requires exact match of both intents and slots at the sentence level; partial matches do not count.
- Multiple intent detection requires predicting the exact set of intents; missing or extra intents count as incorrect.
- Slot F1 is computed at the token level, so boundary mismatches (e.g., BIO tagging errors) heavily penalize the score.

## Evidence (verbatim from paper)

> As for evaluation metrics, following previous works, we adopt accuracy (Acc) for multiple intent detection, F1 score for slot filling, and overall accuracy for the sentence-level semantic frame parsing. Overall accuracy denotes the ratio of sentences whose intents and slots are all correctly predicted.

## Citation

```bibtex
@misc{xing2022coguidingnet,
  title={Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs},
  author={Xing et al. (2022)},
  year={2022},
  note={arXiv:2210.10375}
}
```

- arXiv: 2210.10375