Mbib Media Bias Eval

---
name: mbib-media-bias-eval
description: This evaluation probes a model's ability to detect various forms of media bias in text, including racial, gender, hate speech, fake news, cognitive, and text-level context bias. It tests whether the model can identify subtle, context-dependent linguistic cues and stereotypes across diverse news and social media samples. Use when the user wants to benchmark on Media Bias Identification Benchmark (MBIB), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.20158
  bibtex_key: wen2024chatgptvsmediabias
  confidence: high
---

# mbib-media-bias-eval

> ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models — Wen et al. (2024) (arXiv:2403.20158, 2024)

## What this evaluates

This evaluation probes a model's ability to detect various forms of media bias in text, including racial, gender, hate speech, fake news, cognitive, and text-level context bias. It tests whether the model can identify subtle, context-dependent linguistic cues and stereotypes across diverse news and social media samples.

## Datasets

- **Media Bias Identification Benchmark (MBIB)** — total ?; splits: train (123123), test (31200)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard binary classification accuracy: the proportion of correctly predicted bias labels (1 for presence, 0 for absence) out of the total test instances.

## Input / output format

**Input**: A text snippet (news article, social media post, or statement) accompanied by a task-specific system prompt instructing the model to identify a specific type of bias (e.g., racial, gender, hate speech).

**Output**: A JSON object containing a single 'bias' key with a value of 1 (bias present) or 0 (bias absent).

## Scoring recipe

```python
correct = 0
total = 0
for pred_json, gold_label in zip(predictions, gold_labels):
    pred_label = pred_json['bias']
    if pred_label == gold_label:
        correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- Prompt dependency: ChatGPT's performance is highly sensitive to the exact prompt used, which was selected via a small 60-example validation set rather than a rigorous prompt-tuning protocol.
- Label binarization: Continuous labels in the original MBIB are binarized using dataset-specific thresholds recommended by the original authors, which may not align with standard binary classification conventions or the model's natural decision boundary.
- Subsampling bias: Large-scale tasks (Cognitive Bias, Hate Speech) were randomly subsampled to 10% of their original size before the 80/20 split, potentially skewing results compared to full-dataset evaluations.

## Evidence (verbatim from paper)

> For datasets with continuous labels, binarization is achieved by defining a threshold, with the author's recommended threshold followed wherever possible. ... The performance of these models on the test datasets serves as a comparison against the performance of ChatGPT.

## Citation

```bibtex
@misc{wen2024chatgptvsmediabias,
  title={ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models},
  author={Wen et al. (2024)},
  year={2024},
  note={arXiv:2403.20158}
}
```

- arXiv: 2403.20158