Dsf Gan Utility Eval

---
name: dsf-gan-utility-eval
description: Evaluates the predictive utility of synthetic tabular data generated by a GAN. It measures how well a downstream classifier or regressor trained on the synthetic samples performs when evaluated on a strictly held-out real validation set. Use when the user wants to benchmark on Two distinct tabular datasets (names in Appendix A), or asks about evaluating this task. Reports model performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.18267
  bibtex_key: perets2024dsfgan
  confidence: medium
---

# dsf-gan-utility-eval

> DSF-GAN: DownStream Feedback Generative Adversarial Network — Perets et al. (2024) (arXiv:2403.18267, 2024)

## What this evaluates

Evaluates the predictive utility of synthetic tabular data generated by a GAN. It measures how well a downstream classifier or regressor trained on the synthetic samples performs when evaluated on a strictly held-out real validation set.

## Datasets

- **Two distinct tabular datasets (names in Appendix A)** — total ?; splits: train (-1), test (-1)

## Metrics

- `model performance` **(primary)** — range: other
  - Accuracy for classification tasks or RMSE/MAE for regression tasks, computed by training a logistic or linear model on synthetic data and evaluating it on a held-out real validation set.

## Input / output format

**Input**: Real tabular dataset for GAN training and a set-aside real validation set for evaluation.

**Output**: Synthetic tabular samples generated by the trained GAN, used to train a downstream model.

## Scoring recipe

```python
# Train GAN for N epochs with downstream feedback loss
# Sample n synthetic samples from trained GAN
synthetic_X, synthetic_y = G.sample(n)
# Train downstream model on synthetic data
model = LogisticRegression() if classification else LinearRegression()
model.fit(synthetic_X, synthetic_y)
# Evaluate on held-out real validation set
val_X, val_y = get_held_out_real_validation_set()
metric_value = model.score(val_X, val_y)
```

## Common pitfalls

- Using synthetic data for validation instead of a strictly held-out real validation set.
- Ambiguity in metric selection: classification tasks require accuracy/F1 while regression requires RMSE/MAE, but the paper only states 'model performance'.
- Failing to exclude the validation set from the GAN's training data, leading to data leakage.

## Evidence (verbatim from paper)

> Post-training, we sampled $n$ samples from the trained model, and used it as a training set for a regression or classification model, we evaluated the model performance using a set-aside validation set comprised of real samples which were excluded from the GAN training.

## Citation

```bibtex
@misc{perets2024dsfgan,
  title={DSF-GAN: DownStream Feedback Generative Adversarial Network},
  author={Perets et al. (2024)},
  year={2024},
  note={arXiv:2403.18267}
}
```

- arXiv: 2403.18267