Wikisql Eval

---
name: wikisql-eval
description: Evaluates a model's ability to translate natural language questions into executable SQL queries against a given database schema. It probes semantic parsing, table understanding, and the generation of syntactically and semantically correct structured queries. Use when the user wants to benchmark on WikiSQL, or asks about evaluating this task. Reports Acc_ex.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1709.00103
  bibtex_key: [REDACTED]
  confidence: high
---

# wikisql-eval

> Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning — Zhong et al. (2017) (arXiv:1709.00103, 2017)

## What this evaluates

Evaluates a model's ability to translate natural language questions into executable SQL queries against a given database schema. It probes semantic parsing, table understanding, and the generation of syntactically and semantically correct structured queries.

## Datasets

- **WikiSQL** — total 80654; splits: dev (-1), test (-1); repo https://github.com/salesforce/WikiSQL

## Metrics

- `Acc_lf` — range: percent
  - Logical Form Accuracy: exact string match between the generated SQL query and the ground truth query.
- `Acc_ex` **(primary)** — range: percent
  - Execution Accuracy: the generated query is executed on the database, and its result set is compared to the result set of the ground truth query. Returns 1 if they match, 0 otherwise.

## Input / output format

**Input**: Natural language question paired with the database table schema (column names, types, and table name).

**Output**: A valid SQL SELECT query string.

## Scoring recipe

```python
def score_acc_ex(predictions, golds, db_engine):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            pred_res = db_engine.execute(pred)
            gold_res = db_engine.execute(gold)
            if set(pred_res) == set(gold_res):
                correct += 1
        except:
            pass
    return correct / len(predictions)

def score_acc_lf(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(predictions)
```

## Common pitfalls

- Execution accuracy is order-invariant for WHERE conditions, so models can generate syntactically different but semantically equivalent queries (e.g., swapped AND clauses) and still get full credit.
- Column names with multiple tokens (e.g., 'Miles (km)') are prone to tokenization errors, leading to invalid queries or mismatched column references.
- Rare words in conditions are often hallucinated or truncated by baselines, hurting logical form accuracy even if execution accuracy remains high.

## Evidence (verbatim from paper)

> Dev Acc_lf | Dev Acc_ex | Test Acc_lf | Test Acc_ex

## Citation

```bibtex
@misc{zhong2017seq2sql,
  title={Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning},
  author={Zhong et al. (2017)},
  year={2017},
  note={arXiv:1709.00103}
}
```

- arXiv: 1709.00103