Spider Cosql Eval

---
name: spider-cosql-eval
description: Evaluates the text-to-SQL and dialog state tracking capabilities of language models by measuring how accurately they generate syntactically and semantically valid SQL queries from natural language questions, with and without constrained auto-regressive decoding. Use when the user wants to benchmark on Spider, CoSQL, or asks about evaluating this task. Reports exact-set-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.05093
  bibtex_key: scholak2021picard
  confidence: high
---

# spider-cosql-eval

> PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models — Scholak et al. (2021) (arXiv:2109.05093, 2021)

## What this evaluates

Evaluates the text-to-SQL and dialog state tracking capabilities of language models by measuring how accurately they generate syntactically and semantically valid SQL queries from natural language questions, with and without constrained auto-regressive decoding.

## Datasets

- **Spider** — total 7000; splits: train (7000), dev (-1), test (-1)
- **CoSQL** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `exact-set-match accuracy` **(primary)** — range: percent
  - Compares predicted and ground-truth SQL queries by parsing both into a normalized data structure. Ignores literal query values but can decrease under semantic-preserving rewrites.
- `execution accuracy` — range: percent
  - Compares the results of executing the predicted SQL query against the ground-truth SQL query on the provided database contents. Sensitive to literal values but has a high false positive rate.
- `test-suite execution accuracy` — range: percent
  - Extends execution accuracy by testing against multiple database instances per SQL schema, optimized to reduce false positives and approximate semantic accuracy.
- `question match accuracy` — range: percent
  - Exact-set-match accuracy applied to individual questions within a dialog interaction.
- `interaction match accuracy` — range: percent
  - Joint exact-set-match accuracy over all questions in a multi-turn dialog interaction.

## Input / output format

**Input**: Natural language question concatenated with database schema (and previous dialog questions in reverse chronological order for CoSQL), truncated to 512 tokens. Keywords and identifiers are lowercased.

**Output**: A single SQL query string, with keywords and identifiers converted to lowercase.

## Scoring recipe

```python
def score(predictions, golds, db_contents):
    em_scores, ex_scores = [], []
    for pred, gold in zip(predictions, golds):
        em_scores.append(normalize_sql(pred) == normalize_sql(gold))
        pred_res = execute_sql(pred, db_contents)
        gold_res = execute_sql(gold, db_contents)
        ex_scores.append(pred_res == gold_res)
    return {
        'exact_set_match_accuracy': sum(em_scores) / len(em_scores),
        'execution_accuracy': sum(ex_scores) / len(ex_scores)
    }
```

## Common pitfalls

- Execution accuracy can yield false positives when semantically different SQL queries produce identical results on a specific database instance.
- Exact-set-match accuracy is sensitive to literal query values and semantic-preserving rewrites, potentially underestimating correctness.
- Constrained decoding must be applied incrementally during beam search; applying it only post-hoc significantly reduces effectiveness.

## Evidence (verbatim from paper)

> On Spider, we determine model performance based on three metrics: exact-set-match accuracy, execution accuracy, and test-suite execution accuracy (Zhong et al., 2020). Exact-set-match accuracy compares the predicted and the ground-truth SQL query by parsing both into a normalized data structure. This comparison is not sensitive to literal query values and can decrease under semantic-preserving SQL query rewriting. Execution accuracy compares the results of executing the predicted and ground-truth SQL queries on the database contents shipped with the Spider dataset.

## Citation

```bibtex
@misc{scholak2021picard,
  title={PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models},
  author={Scholak et al. (2021)},
  year={2021},
  note={arXiv:2109.05093}
}
```

- arXiv: 2109.05093