Spider Eval

---
name: spider-eval
description: Evaluates a model's ability to translate natural language questions into correct SQL queries across diverse database domains. It probes schema linking, lexical matching, and complex query synthesis including joins, aggregations, and subqueries. Use when the user wants to benchmark on Spider, or asks about evaluating this task. Reports exact matching accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1905.08205
  bibtex_key: guo2019irnet
  confidence: medium
---

# spider-eval

> Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation — Guo et al. (2019) (arXiv:1905.08205, 2019)

## What this evaluates

Evaluates a model's ability to translate natural language questions into correct SQL queries across diverse database domains. It probes schema linking, lexical matching, and complex query synthesis including joins, aggregations, and subqueries.

## Datasets

- **Spider** — total ?; splits: train (-1), dev (-1); HF `spider_dataset`; repo https://github.com/taoyds/spider

## Metrics

- `exact matching accuracy` **(primary)** — range: [0, 1]
  - A prediction is considered correct if and only if the generated SQL query is exactly identical to the ground truth SQL query, character-for-character. Otherwise, it is marked incorrect.

## Input / output format

**Input**: Natural language question, database schema (table and column names), and schema linking results (entity types and matches).

**Output**: A SemQL intermediate representation query, which is deterministically converted to a SQL query for evaluation.

## Scoring recipe

```python
def exact_match_accuracy(predictions, gold):
    correct = 0
    for pred, gold_sql in zip(predictions, gold):
        if pred.strip() == gold_sql.strip():
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Exact matching requires character-for-character identity, so minor whitespace or case differences cause a penalty unless normalized.
- The evaluation is performed on the final SQL query derived from the intermediate SemQL representation, not on the SemQL itself.
- Cross-domain evaluation means the model must generalize to databases it has never seen during training, relying heavily on schema linking.

## Evidence (verbatim from paper)

> IRNet achieves 46.7% exact matching accuracy on Spider—19.5% higher than prior state-of-the-art—and sets the new leaderboard record, with BERT-augmented performance reaching 54.7%;

## Citation

```bibtex
@misc{guo2019irnet,
  title={Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation},
  author={Guo et al. (2019)},
  year={2019},
  note={arXiv:1905.08205}
}
```

- arXiv: 1905.08205