Bull Eval

---
name: bull-eval
description: Evaluates the ability of LLMs to generate correct SQL queries from natural language questions in financial domains. It tests schema linking, cross-database transfer, and output calibration capabilities specific to fund, stock, and macroeconomic data. Use when the user wants to benchmark on BULL, or asks about evaluating this task. Reports execution accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.10506
  bibtex_key: zhang2024finsql
  confidence: high
---

# bull-eval

> FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis — Zhang et al. (2024) (arXiv:2401.10506, 2024)

## What this evaluates

Evaluates the ability of LLMs to generate correct SQL queries from natural language questions in financial domains. It tests schema linking, cross-database transfer, and output calibration capabilities specific to fund, stock, and macroeconomic data.

## Datasets

- **BULL** — total 4966; splits: train (3966), dev (1000)

## Metrics

- `execution accuracy (EX)` **(primary)** — range: percent
  - Executes both the predicted and ground-truth SQL queries on the target database and returns 1 if the result sets are identical, 0 otherwise. Averaged over all instances.

## Input / output format

**Input**: Natural language question and the relevant database schema (tables and columns).

**Output**: A syntactically valid SQL query string.

## Scoring recipe

```python
def calc_ex(predictions, golds, databases):
    correct = 0
    for pred, gold, db in zip(predictions, golds, databases):
        try:
            pred_res = set(db.execute(pred))
            gold_res = set(db.execute(gold))
            if pred_res == gold_res:
                correct += 1
        except Exception:
            pass
    return correct / len(predictions) * 100
```

## Common pitfalls

- Execution accuracy requires running queries against a live database, not just parsing or string matching.
- Result sets must be compared as unordered collections (e.g., using sets) to account for SQL engines returning rows in arbitrary order.
- GPT-based baselines often exceed context windows, forcing truncation or expensive 32k models, which skews cost and accuracy comparisons.

## Evidence (verbatim from paper)

> We choose execution accuracy (EX) as our evaluation metric, as implemented by Test Suite Accuracy*(Zhong et al., [2020](#bib.bib41 ""))*. This metric is also the official evaluation metric used by the popular Text-to-SQL leaderboard, Spider*(Yu et al., [2018b](#bib.bib38 ""))*. EX executes the predicted SQL query and golden SQL query in the database and judges whether the two have the same execution results.

## Citation

```bibtex
@misc{zhang2024finsql,
  title={FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2401.10506}
}
```

- arXiv: 2401.10506