Momenta Misinformation Eval

---
name: momenta-misinformation-eval
description: Evaluates a multimodal misinformation detection model's ability to classify fake vs. real news across heterogeneous datasets, measuring classification accuracy, ranking quality, and class-balanced performance under calibrated decision thresholds. Use when the user wants to benchmark on Fakeddit, MMCoVaR, Weibo, XFacta, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16172
  bibtex_key: abdollahinejad2026momenta
  confidence: high
---

# momenta-misinformation-eval

> MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection — Abdollahinejad et al. (2026) (arXiv:2604.16172, 2026)

## What this evaluates

Evaluates a multimodal misinformation detection model's ability to classify fake vs. real news across heterogeneous datasets, measuring classification accuracy, ranking quality, and class-balanced performance under calibrated decision thresholds.

## Datasets

- **Fakeddit** — total ?; splits: test (-1)
- **MMCoVaR** — total ?; splits: test (-1)
- **Weibo** — total ?; splits: test (-1)
- **XFacta** — total ?; splits: test (-1)

## Metrics

- `Accuracy` — range: [0, 1]
  - Ratio of correctly classified instances to total instances.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of Precision and Recall: 2 * (Prec * Rec) / (Prec + Rec).
- `Macro-F1` — range: [0, 1]
  - Unweighted mean of F1 scores computed per class, providing class-balanced performance.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring ranking quality across thresholds.
- `MCC` — range: other
  - Matthews Correlation Coefficient, a balanced measure for binary classification especially on imbalanced datasets.

## Input / output format

**Input**: Multimodal social media posts containing paired text and image content.

**Output**: Binary classification label (fake vs. real) or continuous confidence score.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    tp = sum((y_true == 1) & (y_pred == 1))
    tn = sum((y_true == 0) & (y_pred == 0))
    fp = sum((y_true == 0) & (y_pred == 1))
    fn = sum((y_true == 1) & (y_pred == 0))
    acc = (tp + tn) / (tp + tn + fp + fn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1
```

## Common pitfalls

- Failing to calibrate decision thresholds per dataset before test evaluation, which can lead to suboptimal performance on imbalanced splits.
- Relying solely on Accuracy without reporting Macro-F1 or MCC, which masks poor performance on minority classes in imbalanced misinformation datasets.
- Comparing results across datasets without accounting for different score distributions and evaluation protocols used by prior baselines.

## Evidence (verbatim from paper)

> Following standard practice in binary misinformation detection, we report Accuracy, Precision, Recall, F1, Macro-F1, AUC, and MCC. We calibrate decision thresholds on each validation split before final test evaluation to avoid fixing an arbitrary global threshold across datasets with different score distributions

## Citation

```bibtex
@misc{abdollahinejad2026momenta,
  title={MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection},
  author={Abdollahinejad et al. (2026)},
  year={2026},
  note={arXiv:2604.16172}
}
```

- arXiv: 2604.16172