Elliptic Aml Eval

---
name: elliptic-aml-eval
description: Evaluates a graph neural network's ability to classify Bitcoin transactions as licit or illicit using structural and narrative features, while testing a retrieval-augmented generation pipeline for producing regulatory-aligned explanations. Use when the user wants to benchmark on Elliptic AML dataset, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01093
  bibtex_key: khanvilkar2025regulatory
  confidence: high
---

# elliptic-aml-eval

> Regulatory Graphs and GenAI for Real-Time Transaction Monitoring and Compliance Explanation in Banking — Khanvilkar et al. (2025) (arXiv:2506.01093, 2025)

## What this evaluates

Evaluates a graph neural network's ability to classify Bitcoin transactions as licit or illicit using structural and narrative features, while testing a retrieval-augmented generation pipeline for producing regulatory-aligned explanations.

## Datasets

- **Elliptic AML dataset** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Directed transaction edges between address nodes, augmented with structural features (in/out degree, betweenness centrality, transaction frequency) and synthetically generated narrative fields encoded into dense vectors via a fine-tuned DistilBERT model.

**Output**: Binary classification label (licit or illicit) per node. For suspicious transactions, a natural-language explanation generated by GPT-4 aligned with retrieved regulatory clauses.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Temporal split prevents random shuffling or standard cross-validation, making results sensitive to chronological data distribution shifts.
- Synthetic narrative fields may not capture the semantic complexity or noise of real-world financial memo descriptions.
- GPT-4 generation is non-deterministic, so explanation quality cannot be reliably reproduced without fixed seeds or temperature settings.

## Evidence (verbatim from paper)

> The dataset was divided into 80% training and 20% testing, maintaining chronological order to preserve the temporal integrity of the stream. Due to this time-dependent nature, no cross-validation was applied. The system achieved 98.2% F1-score and validated interpretability in expert evaluations.

## Citation

```bibtex
@misc{khanvilkar2025regulatory,
  title={Regulatory Graphs and GenAI for Real-Time Transaction Monitoring and Compliance Explanation in Banking},
  author={Khanvilkar et al. (2025)},
  year={2025},
  note={arXiv:2506.01093}
}
```

- arXiv: 2506.01093