Semanticagent Eval

---
name: semanticagent-eval
description: Evaluates text-to-SQL generation capabilities across cross-domain parsing, knowledge-intensive reasoning, and enterprise-level SQL workflows. It also assesses the semantic validity, execution correctness, and diversity of synthetically generated training data. Use when the user wants to benchmark on Spider, BIRD, Spider2.0, EHRSQL, ScienceBenchmark, Spider-Syn, Spider-Realistic, Spider-DK, or asks about evaluating this task. Reports test-suite accuracy (TS), execution accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21414
  bibtex_key: gao2026semanticagent
  confidence: high
---

# semanticagent-eval

> SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesis — Gao et al. (2026) (arXiv:2604.21414, 2026)

## What this evaluates

Evaluates text-to-SQL generation capabilities across cross-domain parsing, knowledge-intensive reasoning, and enterprise-level SQL workflows. It also assesses the semantic validity, execution correctness, and diversity of synthetically generated training data.

## Datasets

- **Spider** — total 10181; splits: dev (-1), test (-1)
- **BIRD** — total 12751; splits: dev (-1)
- **Spider2.0** — total 135; splits: test (135)
- **EHRSQL** — total ?; splits: (unstated)
- **ScienceBenchmark** — total ?; splits: (unstated)
- **Spider-Syn** — total ?; splits: (unstated)
- **Spider-Realistic** — total ?; splits: (unstated)
- **Spider-DK** — total ?; splits: (unstated)

## Metrics

- `test-suite accuracy (TS)` **(primary)** — range: [0, 1]
  - Fraction of predictions that match the gold execution results across all provided test databases for a given schema.
- `execution accuracy (EX)` **(primary)** — range: [0, 1]
  - Fraction of predictions whose execution results match the gold execution result on a single database.
- `Semantic Alignment (SA)` — range: [0, 1]
  - LLM-based consistency score measuring semantic alignment between the natural language question and the generated SQL query.
- `Successful Execution Rate (SER)` — range: [0, 1]
  - Proportion of synthesized SQL queries that execute without syntax or runtime errors.

## Input / output format

**Input**: Natural language question paired with database schema information (table names, column names, and sampled cell values).

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, db):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if execute_sql(pred, db) == execute_sql(gold, db):
            correct += 1
    return correct / len(predictions)
# For TS, iterate over multiple test databases per schema and require match on all.
```

## Common pitfalls

- Synthetic data contamination: generated examples may accidentally overlap with public evaluation splits; requires n-gram filtering against the eval set.
- Execution vs. semantic correctness: standard EX/TS metrics only verify syntactic/execution equivalence, not whether the SQL actually answers the natural language question semantically.
- Schema-only generation constraint: synthetic data is built exclusively from schema structures and sampled cells without using gold question-SQL pairs, altering the data distribution compared to standard fine-tuning.

## Evidence (verbatim from paper)

> Spider (dev), Spider-Syn, and Spider-Realistic are evaluated with test-suite accuracy (TS), while the remaining benchmarks are evaluated with execution accuracy (EX). EX compares execution results on a single database, whereas TS evaluates predictions over multiple test databases.

## Citation

```bibtex
@misc{gao2026semanticagent,
  title={SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesis},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2604.21414}
}
```

- arXiv: 2604.21414