Esperanto Eval

---
name: esperanto-eval
description: Evaluates the robustness of AI-generated text detectors against back-translation manipulations. It probes whether detectors can maintain high true positive rates when AI-generated text is translated to intermediate languages and back-translated to English, preserving semantics while evading detection. Use when the user wants to benchmark on ESPERANTO, or asks about evaluating this task. Reports True Positive Rate (TPR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.14285
  bibtex_key: ayoobi2024esperanto
  confidence: high
---

# esperanto-eval

> ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination — Ayoobi et al. (2024) (arXiv:2409.14285, 2024)

## What this evaluates

Evaluates the robustness of AI-generated text detectors against back-translation manipulations. It probes whether detectors can maintain high true positive rates when AI-generated text is translated to intermediate languages and back-translated to English, preserving semantics while evading detection.

## Datasets

- **ESPERANTO** — total 792000; splits: full (-1); repo https://github.com/navid-aub/Esperanto-Dataset

## Metrics

- `True Positive Rate (TPR)` **(primary)** — range: [0, 1]
  - TPR = TP / (TP + FN), where TP is the number of AI-generated texts correctly identified as AI, and FN is the number of AI-generated texts incorrectly classified as human. Measures the detector's ability to catch AI text after manipulation.

## Input / output format

**Input**: English text instances (news articles, paper abstracts, Reddit QA responses, product reviews), typically truncated to ~300 words at sentence boundaries. Some instances are back-translated via intermediate languages.

**Output**: Binary classification label (AI-generated vs. Human-written) or detection probability score from the evaluated detector.

## Scoring recipe

```python
def compute_tpr(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    return tp / (tp + fn) if (tp + fn) > 0 else 0.0
```

## Common pitfalls

- Truncating text mid-sentence artificially lowers AI detectability; the dataset enforces sentence-boundary truncation to maintain fairness.
- Back-translation can alter semantics if not carefully combined; the authors use word error rate (WER) to merge back-translated outputs and preserve original meaning.
- Detectors may exhibit domain bias; the dataset spans four distinct writing styles (news, scientific, informal QA, product reviews) to mitigate this.

## Evidence (verbatim from paper)

> The paper introduces back-translation as a novel evasion technique to manipulate AI-generated text, reducing the true positive rate (TPR) of existing AI detection systems by up to 52%—with one detector (RADAR) showing a 52% drop in TPR and a minimal 1.85% TPR decline in a robust countermeasure.

## Citation

```bibtex
@misc{ayoobi2024esperanto,
  title={ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination},
  author={Ayoobi et al. (2024)},
  year={2024},
  note={arXiv:2409.14285}
}
```

- arXiv: 2409.14285