Sede Eval

---
name: sede-eval
description: Evaluates text-to-SQL models on naturally occurring, under-specified user queries from Stack Exchange. Probes the model's ability to handle real-world ambiguity, nested subqueries, parameterized queries, and domain-specific schema knowledge without relying on perfectly specified instructions. Use when the user wants to benchmark on SEDE, or asks about evaluating this task. Reports PCM-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.05006
  bibtex_key: hazoom2021sede
  confidence: high
---

# sede-eval

> Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data — Hazoom et al. (2021) (arXiv:2106.05006, 2021)

## What this evaluates

Evaluates text-to-SQL models on naturally occurring, under-specified user queries from Stack Exchange. Probes the model's ability to handle real-world ambiguity, nested subqueries, parameterized queries, and domain-specific schema knowledge without relying on perfectly specified instructions.

## Datasets

- **SEDE** — total 12023; splits: train (-1), val (-1), test (-1); repo https://github.com/hirupert/sede

## Metrics

- `PCM-F1` **(primary)** — range: [0, 1]
  - F1 score computed over partially matched SQL clauses between predicted and gold queries. Designed to relax exact-match strictness to capture structural similarity in real-world, under-specified queries.
- `PCM-EM` — range: [0, 1]
  - Binary accuracy that returns 1 if and only if PCM-F1 equals 1, otherwise 0.

## Input / output format

**Input**: Natural language utterance (query title), optionally followed by a separator token and a serialized schema listing tables and columns.

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_pcm_f1(pred_sql, gold_sql):
    pred_clauses = extract_clauses(pred_sql)
    gold_clauses = extract_clauses(gold_sql)
    tp = len(match(pred_clauses, gold_clauses))
    precision = tp / len(pred_clauses) if pred_clauses else 0
    recall = tp / len(gold_clauses) if gold_clauses else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

def compute_pcm_em(pred_sql, gold_sql):
    return 1.0 if compute_pcm_f1(pred_sql, gold_sql) == 1.0 else 0.0
```

## Common pitfalls

- Assuming exact string match (EM) is sufficient; real-world queries often differ in column selection or ordering despite high semantic overlap, making PCM-F1 necessary.
- Forgetting that the dataset uses a single schema seen during training, so encoding the schema during inference does not significantly boost performance.
- Including query descriptions in the input; the protocol explicitly uses only titles, which are often under-specified.

## Evidence (verbatim from paper)

> For each experiment we measure PCM-F1 together with a modified version of it, PCM-EM (PCM exact match), that returns an accuracy of 1 for a given prediction if and only if the PCM-F1 value for that prediction is 1.

## Citation

```bibtex
@misc{hazoom2021sede,
  title={Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data},
  author={Hazoom et al. (2021)},
  year={2021},
  note={arXiv:2106.05006}
}
```

- arXiv: 2106.05006