Fraud Detection Eval

---
name: fraud-detection-eval
description: This evaluation probes a machine learning model's ability to accurately detect fraudulent financial transactions in highly imbalanced tabular data, while also measuring the system-level overhead and economic viability of integrating blockchain-based audit trails. It tests both detection accuracy on real-world and synthetic datasets and the practical throughput/latency constraints of on-chain verification workflows. Use when the user wants to benchmark on Kaggle Credit Card Fraud, Enterprise Payment Dataset, or asks about evaluating this task. Reports F1, PR-AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.22096
  bibtex_key: [REDACTED]
  confidence: high
---

# fraud-detection-eval

> Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML — Wang (2026) (arXiv:2604.22096, 2026)

## What this evaluates

This evaluation probes a machine learning model's ability to accurately detect fraudulent financial transactions in highly imbalanced tabular data, while also measuring the system-level overhead and economic viability of integrating blockchain-based audit trails. It tests both detection accuracy on real-world and synthetic datasets and the practical throughput/latency constraints of on-chain verification workflows.

## Datasets

- **Kaggle Credit Card Fraud** — total 284807; splits: 5-fold stratified CV (-1)
- **Enterprise Payment Dataset** — total 10000; splits: 5-fold stratified CV (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Optimized for imbalanced fraud detection where false positives and false negatives carry significant operational costs.
- `PR-AUC` **(primary)** — range: [0, 1]
  - Area under the Precision-Recall curve computed across all classification thresholds. Preferred over ROC-AUC for highly imbalanced datasets (0.17% fraud rate) as it focuses on the positive class performance.
- `Precision` — range: [0, 1]
  - True Positives / (True Positives + False Positives). Measures the proportion of flagged transactions that are actually fraudulent.
- `Recall` — range: [0, 1]
  - True Positives / (True Positives + False Negatives). Measures the proportion of actual frauds correctly identified by the model.

## Input / output format

**Input**: Tabular transaction features representing individual payment requests (e.g., amount, vendor history, approval timing, budget utilization).

**Output**: Binary fraud classification (or fraud probability score) and a SHAP explanation listing the top-5 contributing features for the prediction.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    # PR-AUC computed across thresholds
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred_proba)
    pr_auc = np.trapz(precisions, recalls)
    return {'precision': prec, 'recall': rec, 'f1': f1, 'pr_auc': pr_auc}
```

## Common pitfalls

- PR-AUC is frequently confused with ROC-AUC; the paper explicitly uses PR-AUC due to the extreme class imbalance (0.17% fraud rate), making ROC-AUC misleadingly optimistic.
- Blockchain confirmation latency (3–5 seconds on Polygon PoS) is often overlooked as a system bottleneck, reducing end-to-end throughput to ~200 tx/min compared to ML-only batch throughput (45,000/sec).
- The synthetic Enterprise Payment Dataset lacks real-world enterprise features and is explicitly acknowledged as a limitation; results should not be generalized to production environments without external validation.

## Evidence (verbatim from paper)

> Table II presents detection results using 5-fold stratified cross-validation. Our LightGBM-based detector achieves the highest F1 (0.895) and PR-AUC (0.974).

## Citation

```bibtex
@misc{wang2026tamperproof,
  title={Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML},
  author={Wang (2026)},
  year={2026},
  note={arXiv:2604.22096}
}
```

- arXiv: 2604.22096