Cfdb Eval

---
name: cfdb-eval
description: This benchmark evaluates machine learning models' ability to detect fraudulent customer activity by analyzing aggregated behavioral patterns and transaction features at the customer level. It probes anomaly detection and risk profiling capabilities on synthetic, privacy-compliant financial data with highly imbalanced class distributions. Use when the user wants to benchmark on CFDB (Customer-level Fraud Detection Benchmark), or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.14746
  bibtex_key: jing2024cfdb
  confidence: high
---

# cfdb-eval

> A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation — Jing et al. (2024) (arXiv:2404.14746, 2024)

## What this evaluates

This benchmark evaluates machine learning models' ability to detect fraudulent customer activity by analyzing aggregated behavioral patterns and transaction features at the customer level. It probes anomaly detection and risk profiling capabilities on synthetic, privacy-compliant financial data with highly imbalanced class distributions.

## Datasets

- **CFDB (Customer-level Fraud Detection Benchmark)** — total ?; splits: train (-1), test (-1)

## Metrics

- `Precision` — range: [0, 1]
  - Measures the accuracy of positive predictions, i.e., the proportion of predicted fraudulent transactions that were actually fraudulent.
- `Recall` — range: [0, 1]
  - Also known as sensitivity, it measures the ability of the model to detect all relevant instances, i.e., the proportion of actual fraudulent transactions that were correctly identified by the model.
- `Accuracy` — range: [0, 1]
  - Provides a general indication of the model's ability to correctly label both fraudulent and non-fraudulent transactions.
- `AUC` — range: [0, 1]
  - Represents the area under the ROC curve and provides an aggregate measure of performance across all possible classification thresholds.
- `F1 Score` **(primary)** — range: [0, 1]
  - The harmonic mean of precision and recall, providing a single score that balances both the concerns of precision and recall in one number.

## Input / output format

**Input**: Customer-level aggregated profiles containing behavioral patterns, transaction features, and network structures derived from underlying transaction-level datasets.

**Output**: Binary classification label indicating whether a customer profile is fraudulent or legitimate, along with predicted probabilities for threshold-independent metrics.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, y_prob=None):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = len(y_true) - tp - fp - fn
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    accuracy = (tp + tn) / len(y_true)
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(y_true, y_prob) if y_prob is not None else None
    return {'precision': precision, 'recall': recall, 'accuracy': accuracy, 'f1': f1, 'auc': auc}
```

## Common pitfalls

- The dataset is highly imbalanced, but the authors explicitly state they did not use oversampling methods like SMOTE, which may limit model performance.
- Accuracy is reported but noted as potentially misleading due to class imbalance, so relying solely on it can overstate model effectiveness.
- All models use default hyperparameters without tuning, which may disadvantage more complex architectures like Neural Networks.

## Evidence (verbatim from paper)

> To assess the performance of each model on the CFDB, we employed a variety of evaluation metrics that provide a comprehensive view of each model's effectiveness in detecting fraudulent transactions: Precision: Measures the accuracy of positive predictions, i.e., the proportion of predicted fraudulent transactions that were actually fraudulent. Recall: Also known as sensitivity, it measures the ability of the model to detect all relevant instances, i.e., the proportion of actual fraudulent transactions that were correctly identified by the model. F1 Score: The harmonic mean of precision and recall, providing a single score that balances both the concerns of precision and recall in one number.

## Citation

```bibtex
@misc{jing2024cfdb,
  title={A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation},
  author={Jing et al. (2024)},
  year={2024},
  note={arXiv:2404.14746}
}
```

- arXiv: 2404.14746