Miragenews Eval

---
name: miragenews-eval
description: Evaluates the ability of models to detect AI-generated news content by analyzing multimodal image-caption pairs. It specifically probes robustness to out-of-distribution generators (e.g., DALL-E 3, SDXL) and publishers (e.g., BBC, CNN) compared to in-domain training data. Use when the user wants to benchmark on MiRAGeNews, or asks about evaluating this task. Reports F-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.09045
  bibtex_key: huang2024miragenews
  confidence: high
---

# miragenews-eval

> MiRAGeNews: Multimodal Realistic AI-Generated News Detection — Huang et al. (2024) (arXiv:2410.09045, 2024)

## What this evaluates

Evaluates the ability of models to detect AI-generated news content by analyzing multimodal image-caption pairs. It specifically probes robustness to out-of-distribution generators (e.g., DALL-E 3, SDXL) and publishers (e.g., BBC, CNN) compared to in-domain training data.

## Datasets

- **MiRAGeNews** — total 12500; splits: train (-1), val (-1), test (-1); repo https://github.com/nosna/miragenews

## Metrics

- `F-1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used to balance false positives and false negatives in binary authenticity classification.

## Input / output format

**Input**: Image-caption pairs (multimodal), or individual images (image-only), or captions (text-only) for binary authenticity classification.

**Output**: Binary label indicating whether the content is real (human-authored) or AI-generated.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Zero-shot multimodal LLMs often perform poorly on in-domain data due to training data distribution shifts.
- High precision with extremely low recall can mask poor detection performance, resulting in very low F-1 scores.
- Domain shift from training data (e.g., bedroom images for DIRE) to testing data (news images) severely degrades out-of-domain generalization.

## Evidence (verbatim from paper)

> While the models fine-tuned on ID data have substantially lower performance on DALL-E, we are surprised to find that DIRE FT has a higher average F-1 on SDXL (70.5%) than Midjourney (64.4%).

## Citation

```bibtex
@misc{huang2024miragenews,
  title={MiRAGeNews: Multimodal Realistic AI-Generated News Detection},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2410.09045}
}
```

- arXiv: 2410.09045