Openturingbench Eval

---
name: openturingbench-eval
description: Evaluates the capability of models to detect machine-generated text and attribute it to specific authors or models across diverse scenarios, including mixed human-machine text, out-of-domain content, and outputs from unseen LLMs. Use when the user wants to benchmark on OpenTuringBench, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.11369
  bibtex_key: lacava2025openturingbench
  confidence: high
---

# openturingbench-eval

> OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution — La Cava et al. (2025) (arXiv:2504.11369, 2025)

## What this evaluates

Evaluates the capability of models to detect machine-generated text and attribute it to specific authors or models across diverse scenarios, including mixed human-machine text, out-of-domain content, and outputs from unseen LLMs.

## Datasets

- **OpenTuringBench** — total ?; splits: train (-1), test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Computed from the confusion matrix where 'MACHINE' is the positive class. Uses a weighted-average approach across classes/tasks to account for class size variations, aligning with MGTBench defaults.

## Input / output format

**Input**: Text snippets (human-written, machine-generated, or mixed) with associated metadata for authorship/model attribution tasks.

**Output**: Binary classification label ('HUMAN' or 'MACHINE') for detection, or author/model identifier for attribution.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 'MACHINE' and g == 'MACHINE')
    fp = sum(1 for p, g in zip(predictions, gold) if p == 'MACHINE' and g != 'MACHINE')
    fn = sum(1 for p, g in zip(predictions, gold) if p != 'MACHINE' and g == 'MACHINE')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return f1
```

## Common pitfalls

- Detectors must be fine-tuned for 10 epochs on the OpenTuringBench train set before evaluation to ensure fair comparison, especially for authorship attribution.
- The 'MACHINE' class is strictly treated as the positive class for all metric calculations, which reverses standard human-positive conventions.
- Evaluation uses a weighted-average approach across classes/tasks rather than macro-averaging to handle inherent class imbalances.

## Evidence (verbatim from paper)

> We assessed the performance of OTBDetector and competing methods through standard metrics derived from the confusion matrices obtained in the various TT and AA tasks (cf. Sect. 2.3). These include precision (P), recall (R), and F1-score (F1), where outcomes 'MACHINE' are regarded as instances of the positive class. The scores were computed using a weighted-average approach, which is commonly used to account for variations in class sizes and to align with the default settings in MGTBench.

## Citation

```bibtex
@misc{lacava2025openturingbench,
  title={OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution},
  author={La Cava et al. (2025)},
  year={2025},
  note={arXiv:2504.11369}
}
```

- arXiv: 2504.11369