Titant Fraud Detection Eval

---
name: titant-fraud-detection-eval
description: Evaluates the ability of machine learning models to detect fraudulent financial transactions in real-time using aggregated transaction network features and basic attributes. It probes how well different feature engineering and classification approaches handle severe label imbalance and temporal data splits. Use when the user wants to benchmark on Ant Financial Transaction Dataset, or asks about evaluating this task. Reports F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.07407
  bibtex_key: cao2019titant
  confidence: high
---

# titant-fraud-detection-eval

> TitAnt: Online Real-time Transaction Fraud Detection in Ant Financial — Cao et al. (2019) (arXiv:1906.07407, 2019)

## What this evaluates

Evaluates the ability of machine learning models to detect fraudulent financial transactions in real-time using aggregated transaction network features and basic attributes. It probes how well different feature engineering and classification approaches handle severe label imbalance and temporal data splits.

## Datasets

- **Ant Financial Transaction Dataset** — total ?; splits: train (-1), test (-1), network_build (-1)

## Metrics

- `F1 Score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used as the headline metric for daily performance comparison across eleven configurations.
- `Recall@top 1%` — range: [0, 1]
  - Recall calculated on the top 1% of transactions ranked by predicted fraud probability. Measures the classifier's ability to identify the most suspicious fraud cases.

## Input / output format

**Input**: Transaction records represented by 52 basic features, optionally concatenated with 32-dimensional user node embeddings learned from a transaction network (via DeepWalk or Supervised Node2Vec). Features are discretized into bins for rule-based models.

**Output**: Binary fraud prediction (fraud/non-fraud) or continuous fraud probability score used for ranking and threshold-based evaluation.

## Scoring recipe

```python
def compute_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_rec_top1(scores, gold):
    ranked = np.argsort(-scores)
    top_k = int(len(gold) * 0.01)
    top_indices = ranked[:top_k]
    return sum(1 for i in top_indices if gold[i] == 1) / sum(gold)
```

## Common pitfalls

- Labels are not available in real-time for online testing, so evaluation strictly uses a T+1 offline training/next-day testing split rather than standard random or chronological splits.
- Rule-based models (ID3, C5.0) require data discretization into bins before training, which significantly impacts their performance compared to continuous models.
- Severe label imbalance means supervised embedding methods (S2V) can underperform unsupervised ones (DW) despite using label information.

## Evidence (verbatim from paper)

> In this section, we empirically evaluate the effectiveness of our proposed system for the transaction fraud detection task. Eleven configurations are tested in Table 1 from April 10 to April 16, where F1 score is chosen as the evaluation metric.

## Citation

```bibtex
@misc{cao2019titant,
  title={TitAnt: Online Real-time Transaction Fraud Detection in Ant Financial},
  author={Cao et al. (2019)},
  year={2019},
  note={arXiv:1906.07407}
}
```

- arXiv: 1906.07407