Neural News Detection Eval

---
name: neural-news-detection-eval
description: Evaluates the ability of classifiers and LLMs to detect machine-generated news headlines across four languages. It probes cross-lingual generalization, robustness to zero-shot vs fine-tuned generators, and the effectiveness of linguistic vs transformer-based features for authenticity verification. Use when the user wants to benchmark on Multilingual Neural News Detection Benchmark, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.10724
  bibtex_key: uyuk2024headlines
  confidence: medium
---

# neural-news-detection-eval

> Crafting Tomorrow's Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian — Cem Üyük et al. (arXiv:2408.10724, 2024)

## What this evaluates

Evaluates the ability of classifiers and LLMs to detect machine-generated news headlines across four languages. It probes cross-lingual generalization, robustness to zero-shot vs fine-tuned generators, and the effectiveness of linguistic vs transformer-based features for authenticity verification.

## Datasets

- **Multilingual Neural News Detection Benchmark** — total ?; splits: in-domain test (-1), out-of-domain test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: A single news headline text in English, Turkish, Hungarian, or Persian.

**Output**: Binary classification label indicating whether the headline is human-written or machine-generated (LLM-generated).

## Scoring recipe

```python
def calculate_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- Performance is heavily language-dependent; Persian shows artificially high scores likely due to shorter text lengths.
- Models fine-tuned on in-domain generators often fail to generalize to out-of-domain zero-shot generators.
- LLMs struggle to detect their own generated texts despite high performance on other models.

## Evidence (verbatim from paper)

> Notably, for GPT-4 in English, Random Forest once again achieves the highest F1 score.

## Citation

```bibtex
@misc{uyuk2024headlines,
  title={Crafting Tomorrow's Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian},
  author={Cem Üyük et al.},
  year={2024},
  note={arXiv:2408.10724}
}
```

- arXiv: 2408.10724