Dataco Fraud Detection Eval

---
name: dataco-fraud-detection-eval
description: Evaluates semi-supervised anomaly detection models for supply chain fraud under severe class imbalance and limited label availability. Probes the ability to leverage unsupervised pre-filtering and self-training to improve precision, recall, and F1-score while maintaining low false positive rates. Use when the user wants to benchmark on DataCo Smart Supply Chain Dataset, or asks about evaluating this task. Reports F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06574
  bibtex_key: moradi2025semisupervised
  confidence: high
---

# dataco-fraud-detection-eval

> Semi-Supervised Supply Chain Fraud Detection with Unsupervised Pre-Filtering — Moradi et al. (2025) (arXiv:2508.06574, 2025)

## What this evaluates

Evaluates semi-supervised anomaly detection models for supply chain fraud under severe class imbalance and limited label availability. Probes the ability to leverage unsupervised pre-filtering and self-training to improve precision, recall, and F1-score while maintaining low false positive rates.

## Datasets

- **DataCo Smart Supply Chain Dataset** — total 180519; splits: 10-fold cross-validation (-1)

## Metrics

- `F1-Score` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Optimized for highly imbalanced fraud detection.
- `AUC-ROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `AUC-PR` — range: [0, 1]
  - Area under the Precision-Recall curve, preferred for evaluating performance on highly imbalanced datasets where the positive class is rare.
- `FPR` — range: [0, 1]
  - False Positive Rate: FP / (FP + TN). The protocol targets keeping this below 3.0%.

## Input / output format

**Input**: 52 raw features per transaction (numerical, categorical, temporal) representing supply chain operations across multiple countries.

**Output**: Binary fraud label (fraud vs. legitimate) or anomaly score/confidence for each transaction.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    return {'F1-Score': f1, 'Precision': precision, 'Recall': recall, 'FPR': fpr}
```

## Common pitfalls

- The Isolation Forest contamination factor was set to 0.05 (5%), not the dataset's actual 1.5% fraud rate, to capture borderline anomalies.
- Only 10% of the training portion is labeled per fold, with the remaining 90% treated as unlabeled for self-training, simulating realistic label scarcity.
- Feature selection strictly removed 3 features with >80% missing values and 2 highly correlated features (Pearson r>0.95), resulting in exactly 47 final features.

## Evidence (verbatim from paper)

> Given the class imbalance inherent in fraud detection, we employed multiple evaluation metrics (Precision, Recall, F1-Score, AUC-ROC, AUC-PR, FPR) with statistical significance testing via Wilcoxon signed-rank test. We compared our approach against six baselines... Within each fold of the 10-fold cross-validation, the training portion (approximately 162,467 transactions per fold) was split such that 10% served as labeled data ($\mathcal{D}_{L}$ with 16,247 samples per fold), while the remaining 90% ( 146,220 transactions per fold) formed the unlabeled set $\mathcal{D}_{U}$.

## Citation

```bibtex
@misc{moradi2025semisupervised,
  title={Semi-Supervised Supply Chain Fraud Detection with Unsupervised Pre-Filtering},
  author={Moradi et al. (2025)},
  year={2025},
  note={arXiv:2508.06574}
}
```

- arXiv: 2508.06574