Sql Synth Eval

---
name: sql-synth-eval
description: This benchmark evaluates a model's ability to translate natural language questions into correct SQL queries across diverse, real-world database schemas. It specifically probes the model's capacity to handle complex, multi-operation statements and cross-domain syntax structures that are often underrepresented in traditional benchmarks. Use when the user wants to benchmark on SQL-Synth, or asks about evaluating this task. Reports execution accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.13590
  bibtex_key: wang2025sqlsynth
  confidence: high
---

# sql-synth-eval

> Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation — Wang et al. (2025) (arXiv:2511.13590, 2025)

## What this evaluates

This benchmark evaluates a model's ability to translate natural language questions into correct SQL queries across diverse, real-world database schemas. It specifically probes the model's capacity to handle complex, multi-operation statements and cross-domain syntax structures that are often underrepresented in traditional benchmarks.

## Datasets

- **SQL-Synth** — total ?; splits: train (-1), test (-1)

## Metrics

- `execution accuracy (EX)` **(primary)** — range: [0, 1]
  - Measures whether the predicted SQL query produces the exact same execution results as the corresponding gold SQL query when run against the target database.
- `Data Quality Score` — range: [0, 1]
  - Weighted average of LLM-as-a-judge ratings: (N_e*1 + N_g*0.75 + N_a*0.5 + N_p*0.25) / (N_e + N_g + N_a + N_p), where N_x is the count of samples rated Excellent, Good, Average, or Poor across question, SQL, and result aspects.

## Input / output format

**Input**: Natural language question paired with the target database schema (table names, columns, and relationships).

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_ex(predictions, golds, databases):
    correct = 0
    for pred, gold, db in zip(predictions, golds, databases):
        try:
            pred_res = execute_query(pred, db)
            gold_res = execute_query(gold, db)
            if sets_equal(pred_res, gold_res):
                correct += 1
        except Exception:
            pass
    return correct / len(predictions)
```

## Common pitfalls

- Execution accuracy requires a functional database engine to run queries; string matching or syntax checking alone will yield incorrect scores.
- The dataset intentionally includes complex, multi-operation queries across cross-domain schemas, so models may fail on schema alignment or syntax if evaluated without proper fine-tuning or context handling.
- The quality evaluation relies on GPT-4o as a judge, which may introduce bias or inconsistency compared to ground-truth execution metrics.

## Evidence (verbatim from paper)

> Following previous work, we use execution accuracy (EX) as the evaluation metric, which measures whether the predicted SQL query produces the same execution results as the corresponding gold SQL query.

## Citation

```bibtex
@misc{wang2025sqlsynth,
  title={Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2511.13590}
}
```

- arXiv: 2511.13590