Mimicsql Eval

---
name: mimicsql-eval
description: Evaluates a model's ability to generate correct SQL queries from natural language clinical questions in the healthcare domain. It specifically probes retrieval-based reasoning, robustness to noisy or ambiguous medical terminology, and generalization under data scarcity conditions. Use when the user wants to benchmark on MIMICSQL, or asks about evaluating this task. Reports Execution Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.05569
  bibtex_key: nguyen2026cbrtosql
  confidence: high
---

# mimicsql-eval

> CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain — Nguyen et al. (2026) (arXiv:2603.05569, 2026)

## What this evaluates

Evaluates a model's ability to generate correct SQL queries from natural language clinical questions in the healthcare domain. It specifically probes retrieval-based reasoning, robustness to noisy or ambiguous medical terminology, and generalization under data scarcity conditions.

## Datasets

- **MIMICSQL** — total 10000; splits: train (8000), val (1000), test (1000)

## Metrics

- `Execution Accuracy` **(primary)** — range: [0, 1]
  - Proportion of queries where the executed predicted SQL returns results identical to the executed gold SQL. Formula: $N_{EX}/N$.
- `Logical Form Accuracy` — range: [0, 1]
  - Proportion of queries where the predicted SQL string exactly matches the gold SQL string. Formula: $N_{LF}/N$.
- `Brittleness` — range: [0, 1]
  - Performance drop when top-ranked retrieved cases are stochastically removed based on rank. Formula: $Acc_{original} - Acc_{drop}$.

## Input / output format

**Input**: Natural language clinical question paired with database context (schema and retrieved case templates/examples from either the Complete Database or Incomplete Database environment).

**Output**: A single SQL query string.

## Scoring recipe

```python
def score(predictions, golds, db):
    correct_exec = 0
    correct_lf = 0
    for pred, gold in zip(predictions, golds):
        if execute_sql(pred, db) == execute_sql(gold, db):
            correct_exec += 1
        if pred == gold:
            correct_lf += 1
    acc_ex = correct_exec / len(predictions)
    acc_lf = correct_lf / len(predictions)
    # Brittleness: re-evaluate after rank-based dropout of retrieved cases
    acc_drop = evaluate_with_dropout(predictions, golds, k=5, p_top=1)
    delta_brittle = acc_ex - acc_drop
    return acc_ex, acc_lf, delta_brittle
```

## Common pitfalls

- The Incomplete Database (IDB) setting uses a clustered subset of 774 training examples, not the full test set, which drastically changes retrieval dynamics and generalization expectations.
- Brittleness calculation depends on the retrieval ranking order and a fixed dropout probability schedule ($p_{top}=1$); altering $k$ or the ranking method invalidates the metric.
- Execution accuracy requires an exact schema match and a functional SQLite instance; minor column name or type mismatches cause false negatives.

## Evidence (verbatim from paper)

> Execution Accuracy: $Acc_{\text{EX}}$ measures the proportion of queries that produce correct results: $Acc_{\text{EX}}\=N_{\text{EX}}/N$, where $N_{\text{EX}}$ is the number of queries that return execution results identical to those of the gold queries, and $N$ is the total number of queries. Logical Form Accuracy: $Acc_{\text{LF}}$ measures the proportion of exact SQL matches: $Acc_{\text{LF}}\=N_{\text{LF}}/N$, where $N_{\text{LF}}$ counts exact matches to the gold queries and $N$ is the total number of queries.

## Citation

```bibtex
@misc{nguyen2026cbrtosql,
  title={CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain},
  author={Nguyen et al. (2026)},
  year={2026},
  note={arXiv:2603.05569}
}
```

- arXiv: 2603.05569