Mtop Eval

---
name: mtop-eval
description: Evaluates multilingual task-oriented semantic parsing models on extracting hierarchical intent-slot representations from natural language utterances across multiple languages and transfer settings. It tests the model's ability to generalize across languages using in-language, multilingual, and zero-shot training protocols. Use when the user wants to benchmark on MTOP, Multilingual ATIS, Multilingual TOP, or asks about evaluating this task. Reports exact match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2008.09335
  bibtex_key: li2020mtop
  confidence: high
---

# mtop-eval

> MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark — Li et al. (2020) (arXiv:2008.09335, 2020)

## What this evaluates

Evaluates multilingual task-oriented semantic parsing models on extracting hierarchical intent-slot representations from natural language utterances across multiple languages and transfer settings. It tests the model's ability to generalize across languages using in-language, multilingual, and zero-shot training protocols.

## Datasets

- **MTOP** — total ?; splits: train (-1), val (-1), test (-1)
- **Multilingual ATIS** — total ?; splits: train (-1), val (-1), test (-1)
- **Multilingual TOP** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `exact match` **(primary)** — range: percent
  - A prediction is counted as correct only if the entire predicted intent-slot representation exactly matches the gold representation in structure and tokens. This is a strict exact-match evaluation common in semantic parsing benchmarks.

## Input / output format

**Input**: Natural language utterance in a target language (or English for zero-shot settings).

**Output**: Hierarchical intent-slot representation for MTOP, or flat representation for Multilingual ATIS and Multilingual TOP.

## Scoring recipe

```python
def exact_match(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Alignment models for slot projection are imperfect, especially for low-resource languages, introducing noise and bias into zero-shot training data.
- Multilingual training requires explicit sampling ratios per language to prevent high-resource languages from dominating the batch distribution and degrading performance on low-resource languages.

## Evidence (verbatim from paper)

> reaching 67.2% exact match across 5 languages without target-language data. We conduct thorough experiments on the new dataset we describe in in §3. To further demonstrate the effectiveness of our proposed approaches, we also run additional experiments on the existing multilingual task-oriented semantic parsing datasets including Multilingual ATIS (Upadhyay et al., 2018) and Multilingual TOP (Schuster et al., 2019a).

## Citation

```bibtex
@misc{li2020mtop,
  title={MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark},
  author={Li et al. (2020)},
  year={2020},
  note={arXiv:2008.09335}
}
```

- arXiv: 2008.09335