Spider Syn Eval

---
name: spider-syn-eval
description: This benchmark evaluates the robustness of text-to-SQL models when natural language questions contain real-world synonyms replacing schema-related terms. It probes whether models rely on rigid lexical matching or can generalize to paraphrased queries while preserving the underlying database schema and target SQL query. Use when the user wants to benchmark on Spider, Spider-Syn, or asks about evaluating this task. Reports exact matching accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.01065
  bibtex_key: gan2021towards
  confidence: high
---

# spider-syn-eval

> Towards Robustness of Text-to-SQL Models against Synonym Substitution — Gan et al. (2021) (arXiv:2106.01065, 2021)

## What this evaluates

This benchmark evaluates the robustness of text-to-SQL models when natural language questions contain real-world synonyms replacing schema-related terms. It probes whether models rely on rigid lexical matching or can generalize to paraphrased queries while preserving the underlying database schema and target SQL query.

## Datasets

- **Spider** — total 8034; splits: train (7000), dev (1034)
- **Spider-Syn** — total 8034; splits: train (7000), dev (1034)

## Metrics

- `exact matching accuracy` **(primary)** — range: percent
  - Calculated as the percentage of instances where the model's predicted SQL query exactly matches the ground-truth SQL query. Follows the official Spider evaluation convention which typically normalizes whitespace and case before comparison.

## Input / output format

**Input**: A natural language question paired with the corresponding database schema (table names, column names, data types, and foreign key relationships).

**Output**: A single SQL query string.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_sqls):
    if pred.strip() == gold.strip():
        correct += 1
accuracy = (correct / len(predictions)) * 100
return accuracy
```

## Common pitfalls

- The Spider test set is not publicly accessible, forcing all evaluations to rely on the development set which may not capture full generalization.
- Models frequently fail on synonym-substituted questions due to over-reliance on exact lexical matching between question tokens and schema items, rather than logical reasoning.
- Adversarial training or multi-annotation defenses can show high accuracy on specific attack sets but may degrade performance on clean, in-distribution data.

## Evidence (verbatim from paper)

> Models are evaluated using the official exact matching accuracy metric of Spider. Both Spider and Spider-Syn contain 7000 training and 1034 development samples respectively, where there are 146 databases for training and 20 for development.

## Citation

```bibtex
@misc{gan2021towards,
  title={Towards Robustness of Text-to-SQL Models against Synonym Substitution},
  author={Gan et al. (2021)},
  year={2021},
  note={arXiv:2106.01065}
}
```

- arXiv: 2106.01065