Patsql Sql Synthesis Eval

---
name: patsql-sql-synthesis-eval
description: Evaluates the ability of program-by-example (PBE) systems to synthesize correct SQL queries from example input/output tables. It probes query generation accuracy, synthesis speed, and scalability to larger database schemas. Use when the user wants to benchmark on ase13, so-top, so-dev, so-rec, kaggle, or asks about evaluating this task. Reports solve_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.05807
  bibtex_key: takenouchi2020patsql
  confidence: high
---

# patsql-sql-synthesis-eval

> PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns — Takenouchi et al. (2020) (arXiv:2010.05807, 2020)

## What this evaluates

Evaluates the ability of program-by-example (PBE) systems to synthesize correct SQL queries from example input/output tables. It probes query generation accuracy, synthesis speed, and scalability to larger database schemas.

## Datasets

- **ase13** — total 28; splits: test (28); repo https://github.com/NAIST-SE/PATSQL
- **so-top** — total 57; splits: test (57); repo https://github.com/NAIST-SE/PATSQL
- **so-dev** — total 57; splits: test (57); repo https://github.com/NAIST-SE/PATSQL
- **so-rec** — total 51; splits: test (51); repo https://github.com/NAIST-SE/PATSQL
- **kaggle** — total 33; splits: test (33); repo https://github.com/NAIST-SE/PATSQL

## Metrics

- `solve_rate` **(primary)** — range: percent
  - Number of benchmarks where the synthesized query matches the gold query semantics divided by the total number of benchmarks. Expressed as a count or percentage.
- `synthesis_time` — range: seconds
  - Wall-clock time in seconds required for the synthesis algorithm to produce a valid query or reach the timeout limit.

## Input / output format

**Input**: Example input/output tables (I/O tables) containing specified columns and rows. Optional hints may be provided, including constants used in predicates and required aggregation/window functions.

**Output**: A synthesized SQL query string. For top-k variants (PATSQL_5, SCYTHE), a ranked list of up to 5 candidate queries.

## Scoring recipe

```python
def evaluate(predictions, gold_queries, io_tables):
    solved = 0
    for pred, gold in zip(predictions, gold_queries):
        # Verify semantic equivalence on the provided I/O tables
        if verify_semantics(pred, gold, io_tables):
            solved += 1
    return solved / len(gold_queries)
```

## Common pitfalls

- SCYTHE and BaseLine require explicit hints for aggregation/window functions, while PATSQL does not, creating an unfair comparison if hinting protocols are not standardized.
- I/O tables for non-Kaggle benchmarks use extracted columns (avg 4.3) rather than full-scale schemas (avg 12.1), which artificially inflates scalability metrics for methods that struggle with large schemas.
- Semantic equivalence is verified manually by at least two SQL experts, which can introduce subjectivity or inconsistency across benchmarks.

## Evidence (verbatim from paper)

> For the benchmarks other than kaggle, PATSQL solved 134 benchmarks while SCYTHE solved 128 (see Table 2). In particular, PATSQL succeeded in synthesizing 20 queries from I/O tables with a larger number of cells that SCYTHE failed to solve due to scalability issues.

## Citation

```bibtex
@misc{takenouchi2020patsql,
  title={PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns},
  author={Takenouchi et al. (2020)},
  year={2020},
  note={arXiv:2010.05807}
}
```

- arXiv: 2010.05807