Unite Eval

---
name: unite-eval
description: Evaluates text-to-SQL models on compositional generalization, out-of-domain robustness, and schema-question alignment across 18 diverse datasets and 12 domains. It probes the model's ability to handle long-form query decomposition and cross-domain SQL pattern diversity. Use when the user wants to benchmark on UNITE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.16265
  bibtex_key: lan2023unite
  confidence: high
---

# unite-eval

> UNITE: A Unified Benchmark for Text-to-SQL Evaluation — Lan et al. (2023) (arXiv:2305.16265, 2023)

## What this evaluates

Evaluates text-to-SQL models on compositional generalization, out-of-domain robustness, and schema-question alignment across 18 diverse datasets and 12 domains. It probes the model's ability to handle long-form query decomposition and cross-domain SQL pattern diversity.

## Datasets

- **UNITE** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Execution accuracy: the predicted SQL query is executed on the target database, and the output is compared against the gold execution result. A prediction is correct only if the execution results match exactly.

## Input / output format

**Input**: Natural language question paired with the corresponding database schema.

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_result = execute_sql(pred_sql, db)
        gold_result = execute_sql(gold_sql, db)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Prompt engineering and exemplar selection heavily influence in-context learning performance on out-of-domain data.
- Spider benchmark accuracy is a poor proxy for real-world application robustness.
- Constrained decoding is essential for generalizing to novel SQL structures and avoiding illegal generations.

## Evidence (verbatim from paper)

> We report zero-shot and 3-shot performance for Codex, by following prompt design in Rajkumar et al. (2022). For other fine-tuning based models, we report performance with the original public checkpoints trained on Spider dataset. We also re-train T5-3B and RASAT with our UNITE training set... Our main results are shown in Table 2. Firstly, these SOTA models have performance level 70% ~ 80% for Spider, but below 50% on our UNITE benchmark on average.

## Citation

```bibtex
@misc{lan2023unite,
  title={UNITE: A Unified Benchmark for Text-to-SQL Evaluation},
  author={Lan et al. (2023)},
  year={2023},
  note={arXiv:2305.16265}
}
```

- arXiv: 2305.16265