Bird Critic Eval

---
name: bird-critic-eval
description: This benchmark evaluates an LLM's ability to debug and fix real-world, user-reported SQL issues. It probes the model's capacity to identify logical, syntactic, and semantic flaws in existing queries and generate correct, functional SQL replacements across different database dialects and complexity levels. Use when the user wants to benchmark on BIRD-CRITIC, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.18951
  bibtex_key: li2025swe_sql
  confidence: high
---

# bird-critic-eval

> SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications — Li et al. (2025) (arXiv:2506.18951, 2025)

## What this evaluates

This benchmark evaluates an LLM's ability to debug and fix real-world, user-reported SQL issues. It probes the model's capacity to identify logical, syntactic, and semantic flaws in existing queries and generate correct, functional SQL replacements across different database dialects and complexity levels.

## Datasets

- **BIRD-CRITIC** — total ?; splits: BIRD-CRITIC-PG (-1), BIRD-CRITIC-Multi (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - The percentage of test cases where the model successfully resolves the reported SQL issue and produces a correct, functional query. Calculated as (number of correctly resolved issues / total number of issues) * 100.

## Input / output format

**Input**: A user-reported SQL issue description accompanied by a buggy or incorrect SQL query that needs debugging.

**Output**: A corrected SQL query that resolves the reported issue.

## Scoring recipe

```python
def calculate_sr(predictions, gold):
    correct = 0
    for pred, gold_sql in zip(predictions, gold):
        if execute_and_match(pred, gold_sql): # Semantic/functional equivalence check
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Assuming the task is standard SQL generation from natural language; it specifically requires debugging existing buggy queries based on user-reported issues.
- Overlooking dialect-specific syntax variations when evaluating the multi-dialect split, as models trained primarily on one dialect may fail on others.
- Expecting high baseline performance; the benchmark reveals that even state-of-the-art reasoning models achieve only ~33-39% success rates, indicating significant difficulty.

## Evidence (verbatim from paper)

> These models surpass general-purpose counterparts by an average Success Rate (SR) of 6.13 % on PostgreSQL issues and 8.03 % on multi-dialect issues.

## Citation

```bibtex
@misc{li2025swe_sql,
  title={SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2506.18951}
}
```

- arXiv: 2506.18951