Parrot Eval

---
name: parrot-eval
description: This benchmark evaluates an LLM's ability to accurately translate SQL queries across different database systems and dialects. It probes the model's capacity to handle system-specific syntax, semantic equivalence, and edge-case safeguards without relying on superficial string matching. Use when the user wants to benchmark on PARROT, or asks about evaluating this task. Reports Acc_EX.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23338
  bibtex_key: zhou2025parrot
  confidence: high
---

# parrot-eval

> PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation — Zhou et al. (2025) (arXiv:2509.23338, 2025)

## What this evaluates

This benchmark evaluates an LLM's ability to accurately translate SQL queries across different database systems and dialects. It probes the model's capacity to handle system-specific syntax, semantic equivalence, and edge-case safeguards without relying on superficial string matching.

## Datasets

- **PARROT** — total 598; splits: test (598); repo https://github.com/weAIDB/PARROT

## Metrics

- `Acc_EX` **(primary)** — range: [0, 1]
  - Execution Accuracy: 1 if the translated SQL executes successfully on the target database and returns a result set exactly matching the ground truth SQL's output; 0 otherwise.
- `Acc_RES` — range: [0, 1]
  - Result Accuracy: Measures whether the execution of the translated SQL yields results consistent with the ground truth, prioritizing semantic correctness over string similarity.

## Input / output format

**Input**: A source SQL query, the target database system/dialect specification, and detailed problem instructions provided via a well-crafted prompt.

**Output**: A single translated SQL statement written in the target dialect.

## Scoring recipe

```python
def evaluate(predictions, golds, db_conn):
    ex_correct = 0
    res_correct = 0
    for pred_sql, gold_sql in zip(predictions, golds):
        try:
            pred_res = db_conn.execute(pred_sql)
            gold_res = db_conn.execute(gold_sql)
            if set(pred_res) == set(gold_res):
                ex_correct += 1
                res_correct += 1
        except Exception:
            pass
    return ex_correct / len(predictions), res_correct / len(predictions)
```

## Common pitfalls

- Relying on lexical/string similarity instead of actual database execution to verify correctness.
- Failing to handle dialect-specific runtime safeguards (e.g., division-by-zero checks, ROLLUP syntax, or NULL handling) which cause execution failures despite syntactically plausible translations.
- Performance degradation on lengthy queries due to hallucination or lost-in-the-middle effects, requiring careful token management or segmentation strategies.

## Evidence (verbatim from paper)

> We adopt the evaluation metrics (i.e., $Acc_{EX}$ and $Acc_{RES}$ ) defined in Section 4.

## Citation

```bibtex
@misc{zhou2025parrot,
  title={PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2509.23338}
}
```

- arXiv: 2509.23338