Bird Python Eval

---
name: bird-python-eval
description: Evaluates an LLM's ability to generate executable Python code for file-based data retrieval tasks from natural language questions. It probes the model's capacity to handle explicit procedural logic, resolve ambiguous user intent, and correctly apply domain knowledge without relying on implicit database semantics. Use when the user wants to benchmark on BIRD-Python, or asks about evaluating this task. Reports LLM-based Execution Accuracy (EX).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15728
  bibtex_key: hu2026benchmarking
  confidence: high
---

# bird-python-eval

> Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity — Hu et al. (2026) (arXiv:2601.15728, 2026)

## What this evaluates

Evaluates an LLM's ability to generate executable Python code for file-based data retrieval tasks from natural language questions. It probes the model's capacity to handle explicit procedural logic, resolve ambiguous user intent, and correctly apply domain knowledge without relying on implicit database semantics.

## Datasets

- **BIRD-Python** — total ?; splits: Simple (-1), Moderate (-1), Hard (-1); repo https://github.com/1050727345hu-web/Bird-Python

## Metrics

- `LLM-based Execution Accuracy (EX)` **(primary)** — range: percent
  - The percentage of generated code snippets that execute successfully and produce output exactly matching the gold standard result. Evaluated by running the generated code against the provided data files and comparing the execution output to the expected answer.

## Input / output format

**Input**: Database schema (or file structure), natural language question, and external domain knowledge. Prompts include constraints to ensure the generated code is executable and capable of returning valid results.

**Output**: Python code (typically using Pandas) that performs data retrieval/manipulation and prints or returns the final result.

## Scoring recipe

```python
def compute_ex(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            result = execute_code(pred, data_files=gold['data_files'])
            if results_match(result, gold['output']):
                correct += 1
        except Exception:
            pass
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Conflating a model's failure to resolve ambiguous user intent with a lack of code generation capability.
- Using the original BIRD dataset without verification, as it contains noise that artificially deflates evaluation scores.
- Assuming SQL-specific code generation training transfers directly to Pandas-based data analysis tasks.

## Evidence (verbatim from paper)

> We benchmark Text-to-Python paradigm against the traditional Text-to-SQL approach, employing the LLM-based Execution Accuracy (EX) as the primary evaluation metric. Following the standard evaluation protocol of the BIRD benchmark (Li et al., [2024b]), our input prompts incorporate the database schema, the natural language question, and external knowledge.

## Citation

```bibtex
@misc{hu2026benchmarking,
  title={Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.15728}
}
```

- arXiv: 2601.15728