Clinsql Eval

---
name: clinsql-eval
description: Evaluates clinical text-to-SQL capabilities by requiring models to generate executable BigQuery queries that perform multi-table joins, temporal reasoning, and patient-similarity cohort analysis on electronic health record data. Use when the user wants to benchmark on CLINSQL, or asks about evaluating this task. Reports Execution Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.09876
  bibtex_key: shen2026clinsql
  confidence: high
---

# clinsql-eval

> Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL — Shen et al. (2026) (arXiv:2601.09876, 2026)

## What this evaluates

Evaluates clinical text-to-SQL capabilities by requiring models to generate executable BigQuery queries that perform multi-table joins, temporal reasoning, and patient-similarity cohort analysis on electronic health record data.

## Datasets

- **CLINSQL** — total ?; splits: val (-1), test (-1)

## Metrics

- `SQL Score` — range: percent
  - Rubric-based metric specifically designed for clinical text-to-SQL tasks to evaluate query correctness, clinical relevance, and adherence to schema constraints.
- `Execution Score` **(primary)** — range: percent
  - Execution accuracy percentage; measures whether the generated BigQuery query runs successfully against the target database and returns the expected results.

## Input / output format

**Input**: Natural language clinical question and database schema/context (including table structures, column definitions, and optional schema hints).

**Output**: A single executable BigQuery SQL query, typically extracted from a fenced code block.

## Scoring recipe

```python
def compute_sql_score(pred_sql, gold_sql, rubric):
    return rubric.evaluate(pred_sql, gold_sql)

def compute_exec_score(pred_sql, db_context):
    try:
        result = execute_bigquery(pred_sql, db_context)
        return 1.0 if result_matches_gold(result, gold_sql) else 0.0
    except ExecutionError:
        return 0.0

# Aggregate over dataset
sql_acc = mean([compute_sql_score(p, g, rubric) for p, g in predictions])
exec_acc = mean([compute_exec_score(p, ctx) for p, ctx in predictions])
```

## Common pitfalls

- Models frequently fail on first-pass execution, requiring up to two self-refinement rounds with error feedback to succeed.
- Cohort specification errors are common when models replace explicit ICD or itemid constraints with loose keyword heuristics.
- Schema mismatches and omitted required columns often trigger validation checks, lowering the SQL Score.

## Evidence (verbatim from paper)

> We evaluate all models on CLINSQL using rubric-based metrics specifically designed for clinical text-to-SQL tasks. Our primary evaluation metrics are the SQL Score and the Execution Score.

## Citation

```bibtex
@misc{shen2026clinsql,
  title={Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.09876}
}
```

- arXiv: 2601.09876