Cross Domain Text To Sql Eval

---
name: cross-domain-text-to-sql-eval
description: Evaluates the reliability of cross-domain text-to-SQL benchmarks by exposing flaws in automated metrics like execution accuracy and exact set match, and by introducing human-in-the-loop validation to handle schema ambiguity and query equivalence. Use when the user wants to benchmark on Spider, Spider-DK, BIRD, or asks about evaluating this task. Reports Execution Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.18538
  bibtex_key: pourreza2023evaluating
  confidence: high
---

# cross-domain-text-to-sql-eval

> Evaluating Cross-Domain Text-to-SQL Models and Benchmarks — Pourreza et al. (2023) (arXiv:2310.18538, 2023)

## What this evaluates

Evaluates the reliability of cross-domain text-to-SQL benchmarks by exposing flaws in automated metrics like execution accuracy and exact set match, and by introducing human-in-the-loop validation to handle schema ambiguity and query equivalence.

## Datasets

- **Spider** — total ?; splits: dev (1034)
- **Spider-DK** — total ?; splits: dev (-1)
- **BIRD** — total ?; splits: dev (-1)

## Metrics

- `Execution Accuracy` **(primary)** — range: [0, 1]
  - The percentage of generated SQL queries that produce exactly the same result set as the ground truth query when executed against the database.
- `Exact Set Match Accuracy` — range: [0, 1]
  - The percentage of generated queries where the result set matches the ground truth result set exactly, ignoring row order.
- `Human Accuracy` — range: [0, 1]
  - The percentage of queries deemed correct by human annotators who evaluate semantic equivalence and schema compliance, resolving ambiguities that automated metrics miss.

## Input / output format

**Input**: Natural language question, database schema, and database content.

**Output**: SQL query string.

## Scoring recipe

```python
def exec_acc(pred, gold, db):
    return execute_sql(pred, db) == execute_sql(gold, db)
def exact_set_match(pred, gold, db):
    return set(execute_sql(pred, db)) == set(execute_sql(gold, db))
def human_acc(preds, golds, annotators):
    correct = sum(1 for p, g in zip(preds, golds) if annotators_agree_correct(p, g))
    return correct / len(preds)
```

## Common pitfalls

- Overreliance on strict row ordering or LIMIT 1 causes false negatives when multiple valid rows satisfy the condition.
- SQLite-specific syntax and loose typing mask standard SQL compliance issues, leading to false failures when validated against PostgreSQL.
- Schema ambiguity and incorrect database content assumptions create multiple valid SQL interpretations that automated metrics incorrectly reject.

## Evidence (verbatim from paper)

> Table 2 displays both the execution accuracy and the exact set match accuracy for the reference queries from the BIRD, Spider, and Spider-DK benchmarks after our modifications.

## Citation

```bibtex
@misc{pourreza2023evaluating,
  title={Evaluating Cross-Domain Text-to-SQL Models and Benchmarks},
  author={Pourreza et al. (2023)},
  year={2023},
  note={arXiv:2310.18538}
}
```

- arXiv: 2310.18538