Scm Counterfactual Simulation Eval

---
name: scm-counterfactual-simulation-eval
description: Evaluates the fidelity of a particle filter algorithm for generating counterfactual samples from structural causal models by comparing empirical statistics of the generated samples against known ground-truth distributions and correlations. Use when the user wants to benchmark on Synthetic SCM Simulation, or asks about evaluating this task. Reports proportion of unique observations.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.15328
  bibtex_key: [REDACTED]
  confidence: medium
---

# scm-counterfactual-simulation-eval

> Simulating counterfactuals — Karvanen et al. (2023) (arXiv:2306.15328, 2023)

## What this evaluates

Evaluates the fidelity of a particle filter algorithm for generating counterfactual samples from structural causal models by comparing empirical statistics of the generated samples against known ground-truth distributions and correlations.

## Datasets

- **Synthetic SCM Simulation** — total ?; splits: (unstated); repo https://github.com/JuhaKarvanen/simulatingCounterfactuals

## Metrics

- `proportion of unique observations` **(primary)** — range: [0, 1]
  - Number of unique observations in the generated sample divided by the total sample size n.
- `standardized mean and standard deviation` — range: other
  - Mean and standard deviation of a standardized observed variable z = (x - true_mean) / true_std. Expected values are 0 and 1 respectively.
- `Kolmogorov-Smirnov statistic` — range: [0, 1]
  - Maximum absolute difference between the true cumulative distribution function and the empirical cumulative distribution function of an observed variable.
- `correlation coefficient` — range: [-1, 1]
  - Pearson correlation between two arbitrarily chosen observed variables, compared against the true correlation coefficient.

## Input / output format

**Input**: SCM parameters (number of observed variables, average neighbors, average unobserved confounders, coefficient distributions), number of conditioning variables, and algorithm sample size n.

**Output**: A synthetic sample of size n drawn from the counterfactual distribution using the proposed particle filter algorithm.

## Scoring recipe

```python
def evaluate(samples, true_params):
    n = len(samples)
    unique_prop = len(np.unique(samples, axis=0)) / n
    z = (samples[:, 0] - true_params['true_mean']) / true_params['true_std']
    mean_z = np.mean(z)
    std_z = np.std(z)
    # KS statistic approximation
    x_grid = np.linspace(np.min(samples[:, 0]), np.max(samples[:, 0]), 1000)
    emp_cdf = np.mean(samples[:, 0, None] <= x_grid, axis=0)
    ks_stat = np.max(np.abs(true_params['true_cdf'](x_grid) - emp_cdf))
    corr_emp = np.corrcoef(samples[:, 0], samples[:, 1])[0, 1]
    return {'unique_prop': unique_prop, 'mean_z': mean_z, 'std_z': std_z, 'ks_stat': ks_stat, 'corr_emp': corr_emp}
```

## Common pitfalls

- The univariate and correlation metrics are computed on 'arbitrarily chosen' variables, meaning results may vary depending on which variables are selected in a given simulation run.
- The proportion of unique observations is highly sensitive to the sample size n and whether the underlying SCM variables are discrete or continuous, making cross-experiment comparisons difficult without fixed n.
- The Kolmogorov-Smirnov statistic requires careful handling of the empirical CDF; using too few evaluation points or ignoring ties in discrete data can bias the statistic.

## Evidence (verbatim from paper)

> The performance measures of the simulation experiment evaluate the proportion of unique observations in the sample, univariate statistics of observed variables, and the covariance structure between variables. The proportion of unique observations is calculated by dividing the number of unique observations by the sample size. ... In addition, the Kolmogorov-Smirnov statistic measuring the largest difference between the true cumulative distribution function and the empirical cumulative distribution function is reported. The correlation coefficient between two arbitrarily chosen observed variables was compared to the true correlation coefficient.

## Citation

```bibtex
@misc{karvanen2023simulatingcounterfactuals,
  title={Simulating counterfactuals},
  author={Karvanen et al. (2023)},
  year={2023},
  note={arXiv:2306.15328}
}
```

- arXiv: 2306.15328