Geo880 Atis Eval

---
name: geo880-atis-eval
description: Evaluates a neural semantic parser's ability to map natural language utterances directly to executable SQL queries. It probes compositional generalization and schema grounding by measuring whether predicted queries return the exact same results as gold queries on a target database. Use when the user wants to benchmark on GEO880, ATIS, or asks about evaluating this task. Reports denotation accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1704.08760
  bibtex_key: iyer2017learning
  confidence: high
---

# geo880-atis-eval

> Learning a Neural Semantic Parser from User Feedback — Iyer et al. (2017) (arXiv:1704.08760, 2017)

## What this evaluates

Evaluates a neural semantic parser's ability to map natural language utterances directly to executable SQL queries. It probes compositional generalization and schema grounding by measuring whether predicted queries return the exact same results as gold queries on a target database.

## Datasets

- **GEO880** — total 880; splits: train (600), test (280)
- **ATIS** — total 5418; splits: train (4473), dev (497), test (448)

## Metrics

- `denotation accuracy` **(primary)** — range: [0, 1]
  - Accuracy computed by executing the predicted SQL query on the target database and checking if the returned result set exactly matches the gold result set.

## Input / output format

**Input**: Natural language utterances, with entities anonymized by replacing them with their corresponding types and rare words (frequency=1) replaced with UNK tokens.

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_denotation_accuracy(predictions, golds, database):
    correct = 0
    for pred_sql, gold_sql in zip(predictions, golds):
        pred_result = execute_sql(pred_sql, database)
        gold_result = execute_sql(gold_sql, database)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Confusing denotation accuracy (result-set matching) with logical form exact match; the paper explicitly uses denotations while some baselines report logical form accuracy.
- Assuming standard train/test splits apply uniformly; GEO880 uses cross-validation on the training set for hyperparameter tuning, while ATIS uses a fixed development set.

## Evidence (verbatim from paper)

> We report test set accuracy of our SQL query predictions by executing them on the target database and comparing the result with the true result. Tables 2 and 3 show test accuracies based on denotations for our model on GEO880 and ATIS respectively, compared with previous work.

## Citation

```bibtex
@misc{iyer2017learning,
  title={Learning a Neural Semantic Parser from User Feedback},
  author={Iyer et al. (2017)},
  year={2017},
  note={arXiv:1704.08760}
}
```

- arXiv: 1704.08760