Masksql Eval

---
name: masksql-eval
description: Evaluates the privacy-preserving text-to-SQL generation capability of LLMs. It measures execution accuracy against ground-truth SQL while quantifying privacy protection through token abstraction recall and adversarial re-identification resistance. Use when the user wants to benchmark on BIRD, or asks about evaluating this task. Reports Execution Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23459
  bibtex_key: abedini2025masksql
  confidence: high
---

# masksql-eval

> MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction — Abedini et al. (2025) (arXiv:2509.23459, 2025)

## What this evaluates

Evaluates the privacy-preserving text-to-SQL generation capability of LLMs. It measures execution accuracy against ground-truth SQL while quantifying privacy protection through token abstraction recall and adversarial re-identification resistance.

## Datasets

- **BIRD** — total ?; splits: dev (300)

## Metrics

- `Execution Accuracy` **(primary)** — range: [0, 1]
  - Proportion of generated SQL queries that produce the same result set as the ground-truth query when executed against the corresponding database.
- `Masking Recall (MR)` — range: [0, 1]
  - Ratio of correctly abstracted tokens to the total number of ground-truth sensitive tokens in the natural language question.
- `Re-identification Score (RI)` — range: [0, 1]
  - Ratio of abstracted tokens that an adversary LLM fails to recover or infer from the abstracted prompt and schema.
- `Token Usage` — range: other
  - Average number of tokens consumed across all LLM calls per query generation.

## Input / output format

**Input**: Natural language question paired with the corresponding database schema (and values), optionally pre-processed through a policy-driven abstraction step to mask sensitive tokens.

**Output**: Generated SQL query string.

## Scoring recipe

```python
def score(predictions, golds, dbs, abstracted_tokens, sensitive_tokens, adversary_outputs):
    # Execution Accuracy
    acc = sum(1 for p, g, db in zip(predictions, golds, dbs) if execute(p, db) == execute(g, db)) / len(predictions)
    # Masking Recall
    mr = len(correctly_abstracted) / len(sensitive_tokens)
    # Re-identification Score
    ri = len([t for t in abstracted_tokens if t not in adversary_outputs]) / len(abstracted_tokens)
    # Token Usage
    tokens = [count_tokens(p) for p in predictions]
    return acc, mr, ri, sum(tokens) / len(tokens)
```

## Common pitfalls

- Execution accuracy relies on database execution semantics rather than exact string matching, which can mask syntactic errors that still produce correct results.
- Privacy metrics are tightly coupled to the specific adversary model (GPT-4.1) and the chosen abstraction policy, limiting direct comparability across different privacy settings.
- Token usage aggregates all intermediate LLM calls (e.g., linking, error correction), which may overstate inference costs compared to single-pass prompting baselines.

## Evidence (verbatim from paper)

> For utility evaluation, we use execution accuracy, as defined in the BIRD benchmark. Efficiency is measured by average token usage per query generation. For privacy, we define two metrics. Masking Recall (MR): This metric is defined as the ratio of correctly abstracted tokens to the total number of ground-truth sensitive tokens in the NL question Q. Higher values indicate more protection of the sensitive tokens, resulting in better privacy. Re-identification Score (RI): This metric captures the proportion of abstracted tokens in the NL question Q′ that cannot be re-identified by an adversary. Specifically, we prompt GPT-4.1 with the abstracted NL question Q′ and schema S′ and instruct it to infer the original tokens. The score is then computed as the ratio of the tokens that cannot be recovered by the LLM to the total number of abstract tokens in Q′.

## Citation

```bibtex
@misc{abedini2025masksql,
  title={MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction},
  author={Abedini et al. (2025)},
  year={2025},
  note={arXiv:2509.23459}
}
```

- arXiv: 2509.23459