Tragesql Eval

---
name: tragesql-eval
description: This benchmark evaluates a model's ability to classify the intention of a natural language question relative to a database schema. It probes whether the model can distinguish between answerable queries, questions requiring external knowledge, ambiguous queries, grammatically invalid non-SQL questions, and questions unrelated to the schema. Use when the user wants to benchmark on TRIAGESQL, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.12634
  bibtex_key: zhang2020tragesql
  confidence: high
---

# tragesql-eval

> Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL — Zhang et al. (2020) (arXiv:2010.12634, 2020)

## What this evaluates

This benchmark evaluates a model's ability to classify the intention of a natural language question relative to a database schema. It probes whether the model can distinguish between answerable queries, questions requiring external knowledge, ambiguous queries, grammatically invalid non-SQL questions, and questions unrelated to the schema.

## Datasets

- **TRIAGESQL** — total ?; splits: train (384859), dev (86409), test (2500); repo https://github.com/chatc/TriageSQL

## Metrics

- `Macro F1` **(primary)** — range: [0, 1]
  - The unweighted mean of the F1 scores computed for each of the five intention classes (Improper, ExtKnow, Ambiguous, Non-SQL, Answerable). F1 for a class is the harmonic mean of its precision and recall.

## Input / output format

**Input**: A pair consisting of a natural language question and a database schema, with a special token separating the question from each schema column.

**Output**: A single class label from the set: {Improper, ExtKnow, Ambiguous, Non-SQL, Answerable}.

## Scoring recipe

```python
def macro_f1(y_true, y_pred, classes):
    f1_scores = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models frequently confuse 'ExtKnow' (external knowledge required) with 'Answerable' questions, especially when the schema contains non-mentioned columns that could theoretically answer the query.
- Ambiguous questions receive very low F1 scores due to inherent difficulty in distinguishing schema/value ambiguity from other classes.
- The dataset is highly imbalanced across classes, so accuracy is misleading; Macro F1 must be used to evaluate performance uniformly across all intention types.

## Evidence (verbatim from paper)

> Table 4 shows the result of the RoBERTa model on the proposed test set, achieving a 60% F1 score on average. Some question types can be classified with high F1 scores, such as Improper questions and Non-SQL questions unanswerable by common SQL grammar. However, it only obtains 17% F1 score on Ambiguous questions.

## Citation

```bibtex
@misc{zhang2020tragesql,
  title={Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2010.12634}
}
```

- arXiv: 2010.12634