Beaver Eval

---
name: beaver-eval
description: Probes LLMs' ability to generate correct SQL queries from natural language questions over complex, enterprise-scale databases. It specifically evaluates handling of high schema complexity, multi-table joins, aggregations, and column-to-table mapping in real-world business contexts. Use when the user wants to benchmark on BEAVER, or asks about evaluating this task. Reports execution accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.02038
  bibtex_key: chen2024beaver
  confidence: high
---

# beaver-eval

> BEAVER: An Enterprise Benchmark for Text-to-SQL — Chen et al. (2024) (arXiv:2409.02038, 2024)

## What this evaluates

Probes LLMs' ability to generate correct SQL queries from natural language questions over complex, enterprise-scale databases. It specifically evaluates handling of high schema complexity, multi-table joins, aggregations, and column-to-table mapping in real-world business contexts.

## Datasets

- **BEAVER** — total 203; splits: test (203)

## Metrics

- `execution accuracy` **(primary)** — range: [0, 1]
  - Percentage of generated SQL statements that produce the exact same result set as the gold SQL when executed against the provided database instances.

## Input / output format

**Input**: A natural language question paired with a database schema (table names, column names, data types) and table instances (row data).

**Output**: A single SQL statement.

## Scoring recipe

```python
def score(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        try:
            pred_result = db.execute(pred_sql)
            gold_result = db.execute(gold_sql)
            if set(pred_result) == set(gold_result):
                correct += 1
        except Exception:
            pass
    return correct / len(predictions)
```

## Common pitfalls

- High schema complexity makes column and instance mapping challenging for models.
- Instance mapping is explicitly not annotated in the dataset due to considerable complexity.
- Models trained on public benchmarks often fail to generalize due to missing enterprise-scale multi-table joins and business-domain complexity.

## Evidence (verbatim from paper)

> Following the standard problem setup of text-to-SQL, the input to an LLM includes a natural language question and a database of tables, and the output is a SQL statement whose execution answers the user’s question.

## Citation

```bibtex
@misc{chen2024beaver,
  title={BEAVER: An Enterprise Benchmark for Text-to-SQL},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.02038}
}
```

- arXiv: 2409.02038