Ambiqt Eval

---
name: ambiqt-eval
description: Evaluates a model's ability to generate diverse, semantically valid SQL queries when a natural language question is ambiguous. It specifically probes whether the model can cover multiple valid interpretations of the same query within a limited set of top-k outputs. Use when the user wants to benchmark on AmbiQT, SPIDER, Kaggle DBQA, or asks about evaluating this task. Reports BothInTopK.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.13659
  bibtex_key: bhaskar2023ambiqtbenchmarking
  confidence: high
---

# ambiqt-eval

> Benchmarking and Improving Text-to-SQL Generation under Ambiguity — Bhaskar et al. (2023) (arXiv:2310.13659, 2023)

## What this evaluates

Evaluates a model's ability to generate diverse, semantically valid SQL queries when a natural language question is ambiguous. It specifically probes whether the model can cover multiple valid interpretations of the same query within a limited set of top-k outputs.

## Datasets

- **AmbiQT** — total ?; splits: test (-1); repo https://github.com/testzer0/AmbiQT
- **SPIDER** — total ?; splits: dev (-1)
- **Kaggle DBQA** — total ?; splits: dev (-1)

## Metrics

- `EitherInTopK` — range: percent
  - Percentage of instances where at least one of the top-5 generated SQL queries achieves an Execution Match with any of the gold SQL queries.
- `BothInTopK` **(primary)** — range: percent
  - Percentage of instances where both valid gold SQL queries are present in the top-5 generated outputs, evaluated via Execution Match. Also referred to as Coverage.

## Input / output format

**Input**: Natural language question paired with the corresponding database schema.

**Output**: Top-5 SQL queries generated by the model.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    either_count = 0
    both_count = 0
    for pred_sqls, gold_sqls in zip(predictions, golds):
        # pred_sqls: list of 5 strings, gold_sqls: list of 2 strings
        matches = [execute(p) == execute(g) for p in pred_sqls for g in gold_sqls]
        if any(matches):
            either_count += 1
        if all(any(execute(p) == execute(g) for p in pred_sqls) for g in gold_sqls):
            both_count += 1
    return either_count / len(golds), both_count / len(golds)
```

## Common pitfalls

- Increasing beam width or the number of outputs often reduces coverage because beam search converges to a single dominant interpretation, producing only lexical variations.
- Standard diversity decoding methods (top-k, nucleus, typical sampling) fail to produce meaningful diversity due to skewed token probability distributions, often yielding identical outputs.
- Models may generate syntactically valid but semantically meaningless SQLs (e.g., nested aggregates like max(avg_age)) that do not match either gold query.

## Evidence (verbatim from paper)

> Evaluation Metrics. We present two types of accuracies (i) EitherInTopK - that checks if either of the gold queries feature in the top-5 outputs (ii) BothInTopK - that checks if both gold queries feature in the top-5. We only report the Execution Match (EXM) accuracies for each.

## Citation

```bibtex
@misc{bhaskar2023ambiqtbenchmarking,
  title={Benchmarking and Improving Text-to-SQL Generation under Ambiguity},
  author={Bhaskar et al. (2023)},
  year={2023},
  note={arXiv:2310.13659}
}
```

- arXiv: 2310.13659