Vilbias Eval

---
name: vilbias-eval
description: This benchmark probes a model's ability to detect framing bias in multimodal news content (text-image pairs) and generate grounded, correct rationales for its decisions. It evaluates both closed-ended classification accuracy and open-ended reasoning quality using an LLM-as-judge protocol. Use when the user wants to benchmark on ViLBias, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.17052
  bibtex_key: raza2024vilbias
  confidence: high
---

# vilbias-eval

> ViLBias: Detecting and Reasoning about Bias in Multimodal Content — Raza et al. (2024) (arXiv:2412.17052, 2024)

## What this evaluates

This benchmark probes a model's ability to detect framing bias in multimodal news content (text-image pairs) and generate grounded, correct rationales for its decisions. It evaluates both closed-ended classification accuracy and open-ended reasoning quality using an LLM-as-judge protocol.

## Datasets

- **ViLBias** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/shainarazavi/VILBias

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly classified text-image pairs (biased vs. not biased) against ground-truth annotations.
- `F1 Score` — range: percent
  - Harmonic mean of precision and recall for the biased class, reported as a percentage.
- `Reasoning Accuracy` — range: percent
  - LLM-judged percentage of rationales that correctly support the ground-truth classification decision.
- `Faithfulness` — range: percent
  - LLM-judged percentage of rationales that are fully supported by the provided text and image evidence.

## Input / output format

**Input**: Text-image pairs representing news media content, optionally accompanied by a prompt requesting a bias label and/or a rationale.

**Output**: Closed-ended: categorical label ('biased' or 'not biased'). Open-ended: categorical label followed by a concise rationale explaining the decision.

## Scoring recipe

```python
def compute_classification_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 'biased' and g == 'biased')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'biased' and g == 'not biased')
    fn = sum(1 for p, g in zip(preds, golds) if p == 'not biased' and g == 'biased')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)
    return accuracy, f1

def compute_reasoning_metrics(rationales, golds, inputs, judge_model='GPT-4o'):
    reasoning_acc = sum(1 for r, g in zip(rationales, golds) if judge_model(f'Does rationale {r} correctly support ground truth {g}?')) / len(golds)
    faithfulness = sum(1 for r, i in zip(rationales, inputs) if judge_model(f'Is rationale {r} fully supported by input {i}?')) / len(golds)
    return reasoning_acc, faithfulness
```

## Common pitfalls

- LLM-as-judge bias: Using GPT-4o to evaluate rationales may favor models with similar prompting styles or over-penalize stylistic differences rather than factual grounding.
- Open-ended vs closed-ended gap: Reasoning accuracy consistently lags classification accuracy by 6–12 points; evaluating only closed-ended metrics overestimates a model's true reasoning capability.
- Cross-validation vs fixed splits: The paper uses both an 80/10/10 split and 5-fold CV; results may vary depending on which protocol is reported or averaged.

## Evidence (verbatim from paper)

> Protocols and metrics. We evaluated models under three regimes: zero-shot, five-shot, and full fine-tuning. The dataset was partitioned into training, validation, and test splits with an 80/10/10 ratio, and we additionally employed 5-fold cross-validation to assess robustness. For classification tasks, we report standard metrics including precision, recall, F1 score, and accuracy. To capture performance on open-ended outputs, we further evaluate reasoning quality using LLM-as-judge assessments, focusing on reasoning accuracy and faithfulness. We used GPT4o as the judge model.

## Citation

```bibtex
@misc{raza2024vilbias,
  title={ViLBias: Detecting and Reasoning about Bias in Multimodal Content},
  author={Raza et al. (2024)},
  year={2024},
  note={arXiv:2412.17052}
}
```

- arXiv: 2412.17052