Behavioral Fraud Patterns Eval

---
name: behavioral-fraud-patterns-eval
description: This benchmark evaluates whether synthetic tabular data generators preserve complex behavioral fraud patterns beyond simple statistical fidelity. It specifically probes temporal inter-event timing, burst-like transaction sequences, shared-infrastructure graph motifs across accounts, and velocity-rule trigger rates. Use when the user wants to benchmark on IEEE-CIS Fraud Detection, Amazon Fraud Dataset, or asks about evaluating this task. Reports P1: IET W1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13125
  bibtex_key: sajja2026synthetictabular
  confidence: high
---

# behavioral-fraud-patterns-eval

> Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals — Sajja (2026) (arXiv:2604.13125, 2026)

## What this evaluates

This benchmark evaluates whether synthetic tabular data generators preserve complex behavioral fraud patterns beyond simple statistical fidelity. It specifically probes temporal inter-event timing, burst-like transaction sequences, shared-infrastructure graph motifs across accounts, and velocity-rule trigger rates.

## Datasets

- **IEEE-CIS Fraud Detection** — total 590540; splits: train (472432), test (118108)
- **Amazon Fraud Dataset** — total 151112; splits: test (151112)

## Metrics

- `P1: IET W1` **(primary)** — range: seconds
  - Wasserstein-1 distance ($W_1$) between real and synthetic inter-event timing distributions for fraud transactions. Computed on a 50/50 real-data split to establish a baseline.
- `P1: within-entity autocorrelation gap` — range: dimensionless
  - Absolute difference in autocorrelation of inter-event times within the same entity between real and synthetic data.
- `P2: active lifetime W1` — range: seconds
  - Wasserstein-1 distance between real and synthetic active lifetime distributions (time from first to last transaction) for fraud entities.
- `P2: burst length W1` — range: dimensionless
  - Wasserstein-1 distance between real and synthetic burst length distributions (transactions within a 5-minute window) for fraud entities.
- `P3: fan-out W1` — range: dimensionless
  - Wasserstein-1 distance between real and synthetic fan-out distributions (number of unique devices/IPs per entity) for fraud entities.
- `P4: mean velocity-rule trigger rate gap` — range: dimensionless
  - Absolute difference in the mean rate of velocity-rule triggers between real and synthetic transaction sequences.

## Input / output format

**Input**: Real tabular transaction records (48 curated behavioral columns) used to train generative models; evaluation receives synthetic tabular rows with externally assigned pseudo-entity IDs.

**Output**: Synthetic tabular rows matching the 48-column schema (TransactionDT, TransactionAmt, isFraud, card4, C1–C14, D1–D15, M1–M9, addr1, addr2, dist1, dist2, P_emaildomain, R_emaildomain), with pseudo-entity group labels assigned post-generation.

## Scoring recipe

```python
def compute_degradation_ratio(predictions, gold, pattern):
    gold_baseline = compute_pattern_metric(gold, pattern)
    synth_score = compute_pattern_metric(predictions, pattern)
    return gold_baseline / synth_score
```

## Common pitfalls

- Row-independent generators cannot natively produce entity identifiers, requiring external pseudo-entity assignment that artificially inflates behavioral fidelity scores.
- Excluding high-cardinality entity columns (e.g., card1) and opaque risk scores (V-columns) during training isolates behavioral coherence but masks downstream utility and scalability failures.
- Unconditional sampling on VAE/GAN architectures causes minority-class (fraud) collapse, necessitating conditional sampling to restore realistic fraud rates before evaluation.

## Evidence (verbatim from paper)

> We evaluate P1, P2, and P4 on this dataset; it is the richest publicly available source of within-entity transaction sequences. We evaluate P3 (shared-infrastructure graph motifs) exclusively on this dataset; the device and IP co-occurrence structure yields high attribute fan-out for fraud-linked entities, providing a strong signal for graph motif evaluation.

## Citation

```bibtex
@misc{sajja2026synthetictabular,
  title={Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals},
  author={Sajja (2026)},
  year={2026},
  note={arXiv:2604.13125}
}
```

- arXiv: 2604.13125