Mgt Detector Robustness Eval

---
name: mgt-detector-robustness-eval
description: Evaluates the robustness of machine-generated text detectors against adversarial perturbations such as editing, paraphrasing, prompting, and co-generation. It measures how well detectors maintain binary classification performance when texts are intentionally modified to evade detection. Use when the user wants to benchmark on News-style MGT dataset, or asks about evaluating this task. Reports TPR@FPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.11638
  bibtex_key: [REDACTED]
  confidence: high
---

# mgt-detector-robustness-eval

> Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks — Wang et al. (2024) (arXiv:2402.11638, 2024)

## What this evaluates

Evaluates the robustness of machine-generated text detectors against adversarial perturbations such as editing, paraphrasing, prompting, and co-generation. It measures how well detectors maintain binary classification performance when texts are intentionally modified to evade detection.

## Datasets

- **News-style MGT dataset** — total 10000; splits: train (8000), val (1000), test (1000)

## Metrics

- `AUC ROC` — range: [0, 1]
  - Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
- `TPR@FPR` **(primary)** — range: [0, 1]
  - True positive rate when the false positive rate is fixed at a specific percentage (e.g., 5%). Under this evaluation setting, it is equivalent to Attack Success Rate (ASR).

## Input / output format

**Input**: Text snippets (News-style), labeled as either human-written or machine-generated, with some instances subjected to adversarial attacks (editing, paraphrasing, prompting, co-generating).

**Output**: Binary detection label (human vs. machine) or a continuous detection score/probability used for threshold-independent evaluation.

## Scoring recipe

```python
def compute_tpr_at_fpr(y_true, y_score, fpr_target=0.05):
    fpr, tpr, _ = roc_curve(y_true, y_score)
    tpr_at_target = np.interp(fpr_target, fpr, tpr)
    return tpr_at_target

def compute_auc_roc(y_true, y_score):
    fpr, tpr, _ = roc_curve(y_true, y_score)
    return auc(fpr, tpr)
```

## Common pitfalls

- Accuracy and F1-score are explicitly excluded because they depend on arbitrary threshold settings for metric-based detectors, causing biased comparisons.
- Attacked scenario metrics are reported as relative percentages compared to unattacked baseline performance, not absolute values.
- TPR@FPR functions as an Attack Success Rate (ASR) metric here, meaning higher values indicate worse detector robustness.

## Evidence (verbatim from paper)

> The metrics we use to evaluate detection performance are binary classification metrics AUC ROC and TPR@FPR. AUC ROC is the area under the receiver operating characteristic curve. TPR@FPR is the true positive rate when the false positive rate is at a specific percentage. Under our setting, it is equivalent to Attack Success Rate (ASR) (Tsai et al., 2019). We mainly show TPR@FPR=5%, and TPR@FPR=10% and =20% are additionally recorded in the Appendix D.2. We do not involve Accuracy and F1-score because those metrics are dependent on the setting of the threshold for metric-based detectors, which could be biased in the comparison.

## Citation

```bibtex
@misc{wang2024stumblingblocks,
  title={Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.11638}
}
```

- arXiv: 2402.11638