Text To Sql Annotation Error Eval

---
name: text-to-sql-annotation-error-eval
description: Evaluates the reliability of text-to-SQL benchmarks by quantifying annotation error rates and measuring how these errors distort agent execution accuracy and leaderboard rankings. Use when the user wants to benchmark on BIRD, Spider 2.0-Snow, or asks about evaluating this task. Reports annotation error rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08778
  bibtex_key: jin2026pervasive
  confidence: high
---

# text-to-sql-annotation-error-eval

> Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards — Jin et al. (2026) (arXiv:2601.08778, 2026)

## What this evaluates

Evaluates the reliability of text-to-SQL benchmarks by quantifying annotation error rates and measuring how these errors distort agent execution accuracy and leaderboard rankings.

## Datasets

- **BIRD** — total ?; splits: dev (-1)
- **Spider 2.0-Snow** — total ?; splits: full (-1)

## Metrics

- `annotation error rate` **(primary)** — range: percent
  - Calculated as (number of identified annotation errors / total evaluated examples) × 100. Errors are classified as semantic mismatches, domain knowledge gaps, or ambiguity in the gold SQL annotation.
- `execution accuracy` — range: [0, 1]
  - Standard text-to-SQL metric measuring the fraction of examples where the model's generated SQL produces the exact same result set as the gold SQL when executed against the database.

## Input / output format

**Input**: Natural language question, database schema/context, and (for error detection) the gold SQL annotation.

**Output**: SQL query.

## Scoring recipe

```python
def compute_annotation_error_rate(errors, total):
    return (errors / total) * 100

def compute_execution_accuracy(predictions, golds, db_executor):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if db_executor.run(pred) == db_executor.run(gold):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Relying on unverified gold annotations without human-in-the-loop auditing, which masks systematic benchmark corruption
- Using hidden test sets for local evaluation instead of the designated development set, violating benchmark protocols
- Ignoring domain-specific SQL syntax variations that cause false negatives in execution matching

## Evidence (verbatim from paper)

> In Section[5], we sample 100 examples from the BIRD Dev set and manually correct annotation errors by applying SAPAR. To answer Q2, we then re-evaluate all 16 open-source agents listed on the BIRD leaderboard on both the original and the corrected versions of this sampled subset. We quantify changes in execution accuracy and the relative ranking among these agents.

## Citation

```bibtex
@misc{jin2026pervasive,
  title={Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2601.08778}
}
```

- arXiv: 2601.08778