Modality Bias Eval

---
name: modality-bias-eval
description: This evaluation probes a model's ability to automatically detect and classify sample-specific modality bias in multimodal misinformation content. It measures how well automated quantification methods align with human judgment regarding whether a sample relies on image-only, text-only, or balanced modalities. Use when the user wants to benchmark on Fakeddit, MMFakeBench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.05883
  bibtex_key: lin2025unveiling
  confidence: high
---

# modality-bias-eval

> Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks — Hehai Lin et al. (arXiv:2511.05883, 2025)

## What this evaluates

This evaluation probes a model's ability to automatically detect and classify sample-specific modality bias in multimodal misinformation content. It measures how well automated quantification methods align with human judgment regarding whether a sample relies on image-only, text-only, or balanced modalities.

## Datasets

- **Fakeddit** — total 680000; splits: eval (300)
- **MMFakeBench** — total ?; splits: eval (300)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of samples where the predicted bias type (Uni-image, Modality-balance, or Uni-text) matches the human annotation.
- `F1 score` — range: percent
  - Macro-averaged F1 score across the three bias classes compared to human annotations, computed per class and then averaged.

## Input / output format

**Input**: Multimodal misinformation samples (image + text) randomly selected from benchmarks, processed through automated analysis frameworks (modality benefit, flow, causal effect) to predict bias type.

**Output**: Per-sample classification into one of three bias categories: Uni-image (UI), Modality-balance (MB), or Uni-text (UT). Aggregated metrics include predicted proportions per category, overall accuracy, and F1 score.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = (correct / len(gold_labels)) * 100
    f1_scores = []
    for cls in ['UI', 'MB', 'UT']:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    f1_macro = (sum(f1_scores) / len(f1_scores)) * 100
    return accuracy, f1_macro
```

## Common pitfalls

- The evaluation subset is small (300 samples per dataset) and randomly selected, which may not capture the full distribution of bias types present in the larger benchmarks.
- The paper reports both per-sample classification metrics (accuracy/F1) and dataset-level proportions, which can be conflated if not carefully distinguished.
- Human ground truth reliability is only summarized via Krippendorff's alpha in the appendix, making it difficult to assess annotation consistency without external lookup.

## Evidence (verbatim from paper)

> We report the predicted proportions of each modality bias type and the percentage that aligns with human judgment. For example, $0.78[85.53]$ denotes that multi-view analysis classifies 0.78 of the samples as modality-balance, and among these samples, 85.53% of the results are consistent with human judgment. We also report the overall accuracy and F1 score.

## Citation

```bibtex
@misc{lin2025unveiling,
  title={Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks},
  author={Hehai Lin et al.},
  year={2025},
  note={arXiv:2511.05883}
}
```

- arXiv: 2511.05883