Ehrscl 2024 Eval

---
name: ehrscl-2024-eval
description: Evaluates a system's ability to translate natural language clinical questions into executable SQL and retrieve accurate results from a specialized electronic health record database. It probes complex temporal reasoning, clinical constraint handling, and semantic equivalence in text-to-SQL generation. Use when the user wants to benchmark on EHRSQL 2024, or asks about evaluating this task. Reports execution accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.01053
  bibtex_key: attrach2025conversational
  confidence: high
---

# ehrscl-2024-eval

> Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis — Al Attrach et al. (2025) (arXiv:2507.01053, 2025)

## What this evaluates

Evaluates a system's ability to translate natural language clinical questions into executable SQL and retrieve accurate results from a specialized electronic health record database. It probes complex temporal reasoning, clinical constraint handling, and semantic equivalence in text-to-SQL generation.

## Datasets

- **EHRSQL 2024** — total ?; splits: test (100)

## Metrics

- `execution accuracy` **(primary)** — range: percent
  - Percentage of questions where the generated SQL correctly executes and returns an answer logically and semantically equivalent to the ground truth. Correctness is determined via manual human evaluation rather than strict string matching.

## Input / output format

**Input**: Natural language clinical questions, optionally prefixed with a fixed temporal context instruction (e.g., 'Set the current time to be “2100-12-31 23:59:00”'), and access to the mimic_iv.sqlite database via MCP.

**Output**: An executable SQL query and a final textual answer/result.

## Scoring recipe

```python
correct = 0
for q, gt in dataset:
    sql, ans = model.predict(q)
    if human_check_semantic_equivalence(ans, gt):
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Failing to inject the fixed 'current time' context string for temporal queries, which breaks benchmark alignment.
- Using automated exact-match or strict SQL execution comparison instead of human semantic equivalence checks, which penalizes valid but differently phrased answers.
- Including non-answerable questions in the evaluation set, which artificially deflates accuracy scores.

## Evidence (verbatim from paper)

> Out of the 100 answerable questions, the M3 system correctly generated the SQL and provided the right answer for 94 of them, yielding a simple execution accuracy of 94%. The reported 94% accuracy was determined through a meticulous human evaluation process. For each of the 100 questions, the final answer generated by the M3 system was manually reviewed and compared against the ground truth answer from the EHRSQL dataset. An answer was deemed correct if it was logically and semantically equivalent to the ground truth, even if the phrasing or presentation differed.

## Citation

```bibtex
@misc{attrach2025conversational,
  title={Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis},
  author={Al Attrach et al. (2025)},
  year={2025},
  note={arXiv:2507.01053}
}
```

- arXiv: 2507.01053