Keyinst Eval

---
name: keyinst-eval
description: This evaluation probes a model's ability to formulate correct SQL queries from natural language questions, specifically focusing on capturing structural semantics like GROUP BY, HAVING, ORDER BY, and set operations. It measures how well prompt engineering techniques or fine-tuning improve SQL generation accuracy across different database schemas and question complexities. Use when the user wants to benchmark on StrucQL, Spider, Bird, or asks about evaluating this task. Reports execution accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.00788
  bibtex_key: liu2024keyinst
  confidence: high
---

# keyinst-eval

> KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL — Liu et al. (2024) (arXiv:2411.00788, 2024)

## What this evaluates

This evaluation probes a model's ability to formulate correct SQL queries from natural language questions, specifically focusing on capturing structural semantics like GROUP BY, HAVING, ORDER BY, and set operations. It measures how well prompt engineering techniques or fine-tuning improve SQL generation accuracy across different database schemas and question complexities.

## Datasets

- **StrucQL** — total ?; splits: (unstated)
- **Spider** — total 9034; splits: train (8000), dev (1034)
- **Bird** — total 12751; splits: dev (-1)

## Metrics

- `execution accuracy (EX)` **(primary)** — range: [0, 1]
  - Compares the execution output of the predicted SQL query with the ground truth SQL query on the same database instance. Returns 1 if outputs match exactly, 0 otherwise, averaged over all instances.

## Input / output format

**Input**: Natural language question, database schema, and optionally few-shot examples or keyword instructions.

**Output**: A single SQL query string.

## Scoring recipe

```python
def compute_ex(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_result = execute_sql(pred_sql, db)
        gold_result = execute_sql(gold_sql, db)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- KeyInst alone does not handle schema linking; combining it with SOTA schema-linking methods is required for full pipeline performance.
- Including low-priority keywords (e.g., AVG, COUNT) in the instruction can hinder LLM performance; only high-priority structural operators should be suggested.
- Execution accuracy requires identical database instances for prediction and ground truth; mismatched schemas or data versions will invalidate results.

## Evidence (verbatim from paper)

> Metrics We use execution accuracy (EX) to evaluate different methods. This metric compares the execution output of the predicted SQL query with that of the ground truth SQL query on same database instances.

## Citation

```bibtex
@misc{liu2024keyinst,
  title={KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2411.00788}
}
```

- arXiv: 2411.00788