Xiyansql Eval

---
name: xiyansql-eval
description: Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances. Use when the user wants to benchmark on Spider, Bird, SQL-Eval, NL2GQL, or asks about evaluating this task. Reports Execution Accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.08599
  bibtex_key: gao2024xiyansql
  confidence: high
---

# xiyansql-eval

> A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL — Gao et al. (2024) (arXiv:2411.08599, 2024)

## What this evaluates

Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances.

## Datasets

- **Spider** — total 1981; splits: test (1981)
- **Bird** — total 1534; splits: dev (1534)
- **SQL-Eval** — total 304; splits: test (304)
- **NL2GQL** — total 288; splits: test (288)

## Metrics

- `Execution Accuracy (EX)` **(primary)** — range: percent
  - Execution Accuracy (EX) compares the results of a predicted SQL query and a reference SQL query executed on a specific database instance. A prediction is marked correct if the execution results match exactly.

## Input / output format

**Input**: Natural language question and database schema (provided as M-Schema, DDL, or MAC-SQL representation including table/column names, data types, primary keys, and example values).

**Output**: A single SQL or GQL query string.

## Scoring recipe

```python
correct = 0
for instance in dataset:
    pred_res = execute_sql(instance.pred_sql, instance.db_id)
    ref_res = execute_sql(instance.ref_sql, instance.db_id)
    if pred_res == ref_res:
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- SQL-Eval provides multiple reference SQL queries per question; the paper uses only the first one as ground truth for metric computation.
- Bird benchmark test set is not publicly available, so evaluations are conducted on the development set instead.
- Schema representation format (M-Schema vs DDL vs MAC-SQL) significantly impacts model performance and must be standardized for fair comparison.

## Evidence (verbatim from paper)

> We use Execution Accuracy (EX) to access the effectiveness of the generated SQL queries. EX compares the results of a predicted SQL query and a reference SQL query executed on a specific database instance.

## Citation

```bibtex
@misc{gao2024xiyansql,
  title={A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2411.08599}
}
```

- arXiv: 2411.08599