Mcwq Eval

---
name: mcwq-eval
description: Evaluates a model's ability to generalize compositionally to unseen syntactic structures and cross-lingual settings in semantic parsing. It measures how well models translate natural language questions into correct SPARQL queries across monolingual and zero-shot cross-lingual scenarios. Use when the user wants to benchmark on MCWQ, or asks about evaluating this task. Reports Exact Match (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.11420
  bibtex_key: wang2023multilingual
  confidence: high
---

# mcwq-eval

> On Evaluating Multilingual Compositional Generalization with Translated Datasets — Wang et al. (2023) (arXiv:2306.11420, 2023)

## What this evaluates

Evaluates a model's ability to generalize compositionally to unseen syntactic structures and cross-lingual settings in semantic parsing. It measures how well models translate natural language questions into correct SPARQL queries across monolingual and zero-shot cross-lingual scenarios.

## Datasets

- **MCWQ** — total ?; splits: MCD_mean (-1), Random (-1)

## Metrics

- `Exact Match (%)` **(primary)** — range: percent
  - Percentage of test instances where the model's predicted SPARQL query exactly matches the ground truth query character-for-character.

## Input / output format

**Input**: Natural language question in English, Japanese, or Chinese.

**Output**: A SPARQL query or logical form representing the semantic parsing of the question.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
    if pred.strip() == gold.strip():
        correct += 1
exact_match_acc = (correct / total) * 100
```

## Common pitfalls

- Ground truth (GT) translated datasets contain noise and semantic distortions from neural machine translation, which can unfairly penalize models compared to rule-based (RBMT) translations.
- Adding auxiliary objectives (e.g., question reconstruction and language prediction) to cross-lingual parsers can dramatically decrease performance on compositional generalization tasks compared to logical-form-only baselines.

## Evidence (verbatim from paper)

> Table 3: Monolingual experiment results: Exact match accuracies in percentage (%) are shown here. We present the model performance on the two translated datasets, which share the English branch. MCD_mean represents the average accuracy across 3 MCD splits, and the detailed results breakdown can be found in Appendix D.1. Random refers to the results on the random split.

## Citation

```bibtex
@misc{wang2023multilingual,
  title={On Evaluating Multilingual Compositional Generalization with Translated Datasets},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2306.11420}
}
```

- arXiv: 2306.11420