Tad Bench Eval

---
name: tad-bench-eval
description: Evaluates the effectiveness of various text embedding models combined with different anomaly detection algorithms for identifying text anomalies. It probes how well embedding-based anomaly detection generalizes across diverse domains (spam, fake news, hate speech) and distinguishes between patterned versus context-dependent anomalies. Use when the user wants to benchmark on Email-Spam, SMS-Spam, COVID-Fake, LIAR2, Hate-Speech, OLID, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.11960
  bibtex_key: cao2025tadbench
  confidence: high
---

# tad-bench-eval

> TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection — Cao et al. (2025) (arXiv:2501.11960, 2025)

## What this evaluates

Evaluates the effectiveness of various text embedding models combined with different anomaly detection algorithms for identifying text anomalies. It probes how well embedding-based anomaly detection generalizes across diverse domains (spam, fake news, hate speech) and distinguishes between patterned versus context-dependent anomalies.

## Datasets

- **Email-Spam** — total ?; splits: (unstated)
- **SMS-Spam** — total ?; splits: (unstated)
- **COVID-Fake** — total ?; splits: (unstated)
- **LIAR2** — total ?; splits: (unstated)
- **Hate-Speech** — total ?; splits: (unstated)
- **OLID** — total ?; splits: (unstated)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve; computed as the integral of the True Positive Rate vs. False Positive Rate across all classification thresholds.

## Input / output format

**Input**: Text instances (emails, SMS messages, news articles, social media posts) processed through a chosen embedding model to produce fixed-dimensional vectors.

**Output**: Continuous anomaly scores or binary predictions for each embedded text instance, aggregated into per-dataset AUROC scores.

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auroc(y_true, y_scores):
    # y_true: binary labels (0=normal, 1=anomalous)
    # y_scores: continuous anomaly scores from detector
    return roc_auc_score(y_true, y_scores)
```

## Common pitfalls

- Assuming larger or more parameter-heavy embeddings always yield better anomaly detection; lightweight models like MINILM often match larger ones on explicit tasks.
- Overlooking the critical interaction between embedding type and detector algorithm; kNN and INNE are robust across tasks, while density-based methods like LOF or iForest struggle in high-dimensional semantic spaces.
- Treating all anomaly types as equivalent; patterned anomalies (spam) cluster distinctly in embedding space, whereas context-dependent ones (hate speech) are dispersed, requiring different evaluation strategies.

## Evidence (verbatim from paper)

> Table [2] summarizes the performance of various anomaly detectors combined with LLM-derived embeddings across different datasets... many embedding-detector combinations achieve high AUC scores, with several exceeding 0.8. Figure 5: Boxplot of AUCROC scores for anomaly detectors on different embeddings across 6 datasets.

## Citation

```bibtex
@misc{cao2025tadbench,
  title={TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2501.11960}
}
```

- arXiv: 2501.11960