Sm3 Text To Query Eval

---
name: sm3-text-to-query-eval
description: Evaluates text-to-query systems across relational, document, and graph database models using four query languages (SQL, MQL, Cypher, SPARQL). It probes the ability of models to translate natural language medical questions into correct, executable database queries using standardized SNOMED-CT aligned synthetic patient data. Use when the user wants to benchmark on SM3-Text-to-Query, or asks about evaluating this task. Reports correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.05521
  bibtex_key: sivasubramaniam2024sm3texttoquery
  confidence: medium
---

# sm3-text-to-query-eval

> SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark — Sivasubramaniam et al. (2024) (arXiv:2411.05521, 2024)

## What this evaluates

Evaluates text-to-query systems across relational, document, and graph database models using four query languages (SQL, MQL, Cypher, SPARQL). It probes the ability of models to translate natural language medical questions into correct, executable database queries using standardized SNOMED-CT aligned synthetic patient data.

## Datasets

- **SM3-Text-to-Query** — total 10000; splits: test (10000); repo https://github.com/jf87/SM3-Text-to-Query

## Metrics

- `correctness` **(primary)** — range: [0, 1]
  - Percentage of generated queries that exactly match the expert-verified gold query. Execution-based equivalence may also be used per standard text-to-query protocols.

## Input / output format

**Input**: Natural language question augmented with parameterized values (e.g., patient IDs, disease descriptions, names) drawn from the synthetic database.

**Output**: A single database query string in one of four target languages: SQL, MQL, Cypher, or SPARQL.

## Scoring recipe

```python
def compute_correctness(predictions, gold_queries):
    correct = 0
    for pred, gold in zip(predictions, gold_queries):
        if normalize_string(pred) == normalize_string(gold):
            correct += 1
    return correct / len(gold_queries)
```

## Common pitfalls

- Models may overfit to template structures or specific parameterized values rather than learning general text-to-query translation.
- Non-answerable questions (15 total) require models to recognize missing information rather than generate a query, which can skew accuracy if not handled separately.
- Cross-model schema differences (relational vs. document vs. graph) mean a single prompt template cannot be used across all four database backends without adaptation.

## Evidence (verbatim from paper)

> Overall, we create 408 template questions... For each question template, we manually develop the corresponding query in SQL, SPARQL, Cypher and MQL. The queries are then verified by a second expert for correctness. For scaling the template questions, we augment them by automatically inserting values such as IDs, descriptions of diseases, and patient names queried from the database. This data augmentation step is fully configurable and can be used to generate enriched and linguistically diverse text/query pairs for arbitrary Synthea databases...

## Citation

```bibtex
@misc{sivasubramaniam2024sm3texttoquery,
  title={SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark},
  author={Sivasubramaniam et al. (2024)},
  year={2024},
  note={arXiv:2411.05521}
}
```

- arXiv: 2411.05521