Tabular Generation Eval

---
name: tabular-generation-eval
description: Evaluates the fidelity, utility, and privacy of synthetic tabular data generated by language models compared to real data and other generative baselines. It measures how well the synthetic distribution matches the original across statistical, downstream utility, and privacy dimensions. Use when the user wants to benchmark on Adult, Default, Shoppers, Magic, Beijing, or asks about evaluating this task. Reports C2ST.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18966
  bibtex_key: long2026selfimproving
  confidence: high
---

# tabular-generation-eval

> Self-Improving Tabular Language Models via Iterative Group Alignment — Long et al. (2026) (arXiv:2604.18966, 2026)

## What this evaluates

Evaluates the fidelity, utility, and privacy of synthetic tabular data generated by language models compared to real data and other generative baselines. It measures how well the synthetic distribution matches the original across statistical, downstream utility, and privacy dimensions.

## Datasets

- **Adult** — total ?; splits: test (-1)
- **Default** — total ?; splits: test (-1)
- **Shoppers** — total ?; splits: test (-1)
- **Magic** — total ?; splits: test (-1)
- **Beijing** — total ?; splits: test (-1)

## Metrics

- `C2ST` **(primary)** — range: [0, 1]
  - Two-sample classification test score computed via SDMetrics; higher values indicate better synthetic data quality matching the real distribution.
- `DA AUC` — range: [0, 1]
  - Distinguishability Attack Area Under the ROC Curve. Values closer to 0.5 indicate stronger indistinguishability; reported as |AUC - 0.5| where lower is better.
- `CDE` — range: [0, 1]
  - Column Density Estimation similarity measuring how well the synthetic data matches the real column distributions; higher is better.
- `PCC` — range: [0, 1]
  - Pairwise Correlation similarity measuring how well the synthetic data preserves pairwise feature correlations; higher is better.
- `MLE` — range: percent
  - Machine Learning Efficiency measuring downstream task performance when training models on synthetic data; higher is better.

## Input / output format

**Input**: Real tabular dataset samples and generated synthetic tabular samples (mixed-type columns).

**Output**: Metric scores (CDE, PCC, Shape & Trend errors, α-precision, β-recall, MLE, DA AUC, C2ST) reported as mean ± std over 10 seeds.

## Scoring recipe

```python
def evaluate(gold_real, pred_synthetic):
    c2st = sdmetrics.tabular.C2ST.compute(gold_real, pred_synthetic)
    da_auc = compute_distinguishability_auc(gold_real, pred_synthetic)
    cde = compute_column_density_estimation_similarity(gold_real, pred_synthetic)
    pcc = compute_pairwise_correlation_similarity(gold_real, pred_synthetic)
    mle = train_and_evaluate_downstream_ml(pred_synthetic)
    return {
        "C2ST": c2st,
        "DA_AUC": abs(da_auc - 0.5),
        "CDE": cde,
        "PCC": pcc,
        "MLE": mle
    }
```

## Common pitfalls

- Confusing the directionality of DA AUC: closer to 0.5 is better, so the paper reports |AUC - 0.5| (lower is better) rather than raw AUC.
- Failing to retrain the distinguishability classifier on equal-sized real and synthetic samples at each iteration, which biases the automated quality signal.
- Reporting raw AUC instead of the transformed |AUC - 0.5| for privacy metrics, leading to incorrect interpretation of indistinguishability.

## Evidence (verbatim from paper)

> Synthetic data quality is evaluated across: (1) Fidelity: Column Density Estimation similarity (CDE, higher = better match), Pairwise Correlation similarity (PCC, higher = better), Shape & Trend errors (lower = better), α-precision, and β-recall; (2) Utility: downstream task performance via Machine Learning Efficiency (MLE); (3) Privacy: Distinguishability Attack (DA) AUC, where values closer to 0.5 indicate stronger indistinguishability (we report |AUC -0.5|, lower = better). C2ST scores are computed via the SDMetrics quality convention where higher indicates better synthetic quality.

## Citation

```bibtex
@misc{long2026selfimproving,
  title={Self-Improving Tabular Language Models via Iterative Group Alignment},
  author={Long et al. (2026)},
  year={2026},
  note={arXiv:2604.18966}
}
```

- arXiv: 2604.18966