Multilingual Fraud Detection Eval

---
name: multilingual-fraud-detection-eval
description: Evaluates the ability of machine learning and transformer models to classify multilingual financial messages as legitimate or fraudulent. It probes handling of code-mixed Bangla-English text, low-resource language features, and structural indicators like URLs and phone numbers. Use when the user wants to benchmark on Financial scams detection dataset, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.11358
  bibtex_key: uddin2026multilingual
  confidence: high
---

# multilingual-fraud-detection-eval

> Multilingual Financial Fraud Detection Using Machine Learning and Transformer Models: A Bangla-English Study — Uddin et al. (2026) (arXiv:2603.11358, 2026)

## What this evaluates

Evaluates the ability of machine learning and transformer models to classify multilingual financial messages as legitimate or fraudulent. It probes handling of code-mixed Bangla-English text, low-resource language features, and structural indicators like URLs and phone numbers.

## Datasets

- **Financial scams detection dataset** — total 523; splits: 5-fold stratified CV (523)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Ratio of correctly classified instances to the total number of instances.
- `Macro-averaged F1 score` — range: percent
  - Unweighted mean of the F1 scores for the scam and ham classes, computed as the harmonic mean of precision and recall per class.
- `PR-AUC` — range: percent
  - Area under the Precision-Recall curve, measuring the model's ranking performance across all decision thresholds.

## Input / output format

**Input**: Multilingual financial messages (text) containing Bangla, English, and code-mixed content.

**Output**: Binary label: 'ham' (legitimate) or 'scam' (fraudulent).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    prec_scam = tp / (tp + fp)
    rec_scam = tp / (tp + fn)
    f1_scam = 2 * prec_scam * rec_scam / (prec_scam + rec_scam)
    prec_ham = tn / (tn + fn)
    rec_ham = tn / (tn + fp)
    f1_ham = 2 * prec_ham * rec_ham / (prec_ham + rec_ham)
    macro_f1 = (f1_scam + f1_ham) / 2
    return accuracy, macro_f1
```

## Common pitfalls

- The dataset is small (523 samples), leading to notable variance across cross-validation folds, especially for the transformer model.
- The transformer exhibits a strong bias toward predicting the scam class, resulting in high recall but elevated false positive rates compared to classical models.
- Code-mixing and limited Bangla financial vocabulary constrain both TF-IDF and multilingual transformer representations.

## Evidence (verbatim from paper)

> All models are evaluated using 5-fold stratified cross-validation to ensure robust performance estimation across different data partitions. We report accuracy, macro-averaged F1 score, and Precision-Recall Area Under Curve (PR-AUC) as primary evaluation metrics.

## Citation

```bibtex
@misc{uddin2026multilingual,
  title={Multilingual Financial Fraud Detection Using Machine Learning and Transformer Models: A Bangla-English Study},
  author={Uddin et al. (2026)},
  year={2026},
  note={arXiv:2603.11358}
}
```

- arXiv: 2603.11358