S3eval Sql Execution Eval

---
name: s3eval-sql-execution-eval
description: Evaluates large language models' ability to perform complex, multi-step reasoning and data dependency tracking by executing SQL queries over synthetic, arbitrarily long tables. It probes contextual reasoning, long-context understanding, and structured data manipulation capabilities beyond traditional benchmarks. Use when the user wants to benchmark on S3Eval, or asks about evaluating this task. Reports SQL execution performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.15147
  bibtex_key: lei2023s3eval
  confidence: high
---

# s3eval-sql-execution-eval

> S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models — Lei et al. (2023) (arXiv:2310.15147, 2023)

## What this evaluates

Evaluates large language models' ability to perform complex, multi-step reasoning and data dependency tracking by executing SQL queries over synthetic, arbitrarily long tables. It probes contextual reasoning, long-context understanding, and structured data manipulation capabilities beyond traditional benchmarks.

## Datasets

- **S3Eval** — total ?; splits: Easy (-1), General (-1); repo https://github.com/lfy79001/S3Eval

## Metrics

- `SQL execution performance` **(primary)** — range: percent
  - Percentage of queries where the model's predicted output exactly matches the ground truth execution result. Computed as (correct predictions / total queries) * 100.

## Input / output format

**Input**: A synthetic table (with arbitrary length and structure) provided in text format, followed by a SQL query to be executed on that table.

**Output**: The exact result of the SQL execution (e.g., a list of values or a single scalar), typically returned in plain text or markdown format.

## Scoring recipe

```python
def compute_performance(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize(pred) == normalize(gold):
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Models may hallucinate table structures or column names not present in the prompt.
- Execution performance requires exact string/format matching; minor formatting differences (e.g., extra whitespace, different list syntax) can cause false negatives.
- Long-context settings (20K-80K tokens) often cause performance dips due to context window limits or attention degradation, not just reasoning failure.

## Evidence (verbatim from paper)

> Table 4: SQL Execution Task Performance on different LLMs. SQL execution is a suitable task because SQL can be intertranslated with an natural question. ChatGPT's performance (markdown) improves from 38.0 to 48.5 when using chain-of-thought prompts.

## Citation

```bibtex
@misc{lei2023s3eval,
  title={S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models},
  author={Lei et al. (2023)},
  year={2023},
  note={arXiv:2310.15147}
}
```

- arXiv: 2310.15147