Synthetic Sql Eval

---
name: synthetic-sql-eval
description: Evaluates how synthetic or human-generated column descriptions impact LLM performance on text-to-SQL tasks, and assesses the quality of LLM-generated descriptions across varying semantic difficulty levels. Use when the user wants to benchmark on BIRD-Bench, or asks about evaluating this task. Reports Mean quality scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.04691
  bibtex_key: wretblad2024synthetic
  confidence: high
---

# synthetic-sql-eval

> Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance — Wretblad et al. (2024) (arXiv:2408.04691, 2024)

## What this evaluates

Evaluates how synthetic or human-generated column descriptions impact LLM performance on text-to-SQL tasks, and assesses the quality of LLM-generated descriptions across varying semantic difficulty levels.

## Datasets

- **BIRD-Bench** — total 1534; splits: dev (1534); HF `bird`

## Metrics

- `Mean quality scores` **(primary)** — range: numeric scale (1-5 or 1-10, exact range not specified)
  - Average rating assigned by human evaluators to generated column descriptions, aggregated across four semantic difficulty levels (Self-Evident, Context-Aided, Ambiguity-Prone, Domain-Dependent).
- `Execution Accuracy` — range: [0, 1]
  - Standard BIRD-Bench metric: proportion of predicted SQL queries that produce the exact same result set as the gold SQL query when executed on the target database.

## Input / output format

**Input**: Database schema, specified column, example data rows, and generation instructions. For text-to-SQL evaluation: natural language question, database schema, and column descriptions (metadata).

**Output**: Natural language column description (for generation task) or SQL query (for text-to-SQL task).

## Scoring recipe

```python
# Description quality scoring:
scores = [human_rating(desc) for desc in generated_descriptions]
mean_quality_scores = sum(scores) / len(scores)

# Text-to-SQL scoring (standard BIRD protocol):
predicted_sql = model.generate(question, schema, descriptions)
execution_result = execute_sql(predicted_sql, database)
is_correct = execution_result == gold_result
execution_accuracy = sum(is_correct) / len(test_set)
```

## Common pitfalls

- Assuming only 'perfect' descriptions improve performance; the paper shows 'superfluous' descriptions also significantly boost text-to-SQL accuracy.
- Overlooking the temperature difference: 0.7 for description generation vs 0.0 for text-to-SQL evaluation.
- Failing to account for the two-step ambiguity detection process, which requires human intervention for domain-dependent columns.

## Evidence (verbatim from paper)

> To study the usefulness of column descriptions for text-to-SQL systems, we first evaluated the same LLMs as in the description generation step in a zero-shot setting on the development set of BIRD-Bench, which contains 1534 natural language questions and SQL query pairs. ... Table 2: Mean quality scores for each model’s generated descriptions across the column difficulty levels in the dataset, as rated by the human evaluators.

## Citation

```bibtex
@misc{wretblad2024synthetic,
  title={Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance},
  author={Wretblad et al. (2024)},
  year={2024},
  note={arXiv:2408.04691}
}
```

- arXiv: 2408.04691