Raid Robustness Eval

---
name: raid-robustness-eval
description: Evaluates the adversarial robustness and transferability of AI-generated image detectors against crafted perturbations. It probes whether detectors can maintain classification accuracy when faced with white-box and black-box evasion attacks across different perturbation budgets. Use when the user wants to benchmark on RAID, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.03988
  bibtex_key: eddoubi2025raid
  confidence: high
---

# raid-robustness-eval

> RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors — Eddoubi et al. (2025) (arXiv:2506.03988, 2025)

## What this evaluates

Evaluates the adversarial robustness and transferability of AI-generated image detectors against crafted perturbations. It probes whether detectors can maintain classification accuracy when faced with white-box and black-box evasion attacks across different perturbation budgets.

## Datasets

- **RAID** — total 96000; splits: test (96000); repo https://github.com/pralab/RAID

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and true positive rate (TPR). Defined as F1 = 2 * (Precision * TPR) / (Precision + TPR).
- `Accuracy` — range: [0, 1]
  - Ratio of correctly predicted samples over the total number of samples, using a classification threshold of 0.5.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, plotting True Positive Rate against False Positive Rate across all classification thresholds.

## Input / output format

**Input**: Center-cropped images with detector-specific preprocessing applied. Images are provided as raw tensors or PNG files to avoid lossy compression artifacts.

**Output**: Binary classification label (real/fake) or continuous probability score. A threshold of 0.5 is applied for accuracy calculation.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred_scores, threshold=0.5):
    y_pred = (y_pred_scores >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    precision = tp / (tp + fp + 1e-8)
    tpr = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * tpr / (precision + tpr + 1e-8)
    accuracy = np.mean(y_pred == y_true)
    auroc = roc_auc_score(y_true, y_pred_scores)
    return {'F1': f1, 'Accuracy': accuracy, 'AUROC': auroc}
```

## Common pitfalls

- Applying lossy compression (e.g., JPEG) to adversarial images degrades perturbation transferability and artificially inflates detector robustness.
- Data drift between the D³ test set and the original training datasets of baseline detectors causes significant performance drops unrelated to adversarial attacks.
- Using a fixed 0.5 threshold for accuracy may mislead evaluation on unbalanced datasets; F1-score or AUROC should be prioritized.

## Evidence (verbatim from paper)

> To evaluate the performance of the detectors, we make use of the following metrics: F1-score. The F1 score measures the harmonic mean of the precision and true positive rate (TPR), which provides a metric capable of reliably computing the model’s performance in the presence of unbalanced class distributions. It is defined as: F1=2×(Precision×TPR)/(Precision+TPR).

## Citation

```bibtex
@misc{eddoubi2025raid,
  title={RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors},
  author={Eddoubi et al. (2025)},
  year={2025},
  note={arXiv:2506.03988}
}
```

- arXiv: 2506.03988