Mbib Political Bias Eval

---
name: mbib-political-bias-eval
description: Evaluates large language models' ability to detect political bias in media text using in-context learning and chain-of-thought prompting. It probes the model's capacity to distinguish biased from unbiased content across diverse textual chunks without fine-tuning. Use when the user wants to benchmark on Media Bias Identification Benchmark (MBIB), or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.00782
  bibtex_key: sar2025navigating
  confidence: medium
---

# mbib-political-bias-eval

> Navigating Nuance: In Quest for Political Truth — Sar et al. (2025) (arXiv:2501.00782, 2025)

## What this evaluates

Evaluates large language models' ability to detect political bias in media text using in-context learning and chain-of-thought prompting. It probes the model's capacity to distinguish biased from unbiased content across diverse textual chunks without fine-tuning.

## Datasets

- **Media Bias Identification Benchmark (MBIB)** — total ?; splits: test (-1)

## Metrics

- `Macro-F1` **(primary)** — range: [0, 1]
  - The unweighted mean of the F1 score computed independently for each class (biased vs. unbiased) and then averaged. It treats all classes equally regardless of their support.

## Input / output format

**Input**: Text snippets from the MBIB dataset, formatted with zero-shot, few-shot, or k-shot Chain-of-Thought prompts instructing the model to identify political bias.

**Output**: A binary label (0 or 1) indicating the presence or absence of political bias. The CoT prompting variant may additionally output brief textual explanations alongside the label.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    
    prec_1 = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec_1 = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_1 = 2 * prec_1 * rec_1 / (prec_1 + rec_1) if (prec_1 + rec_1) > 0 else 0
    
    prec_0 = (len(predictions) - tp - fp) / (len(predictions) - tp)
    rec_0 = (len(predictions) - tp - fn) / (len(predictions) - fn)
    f1_0 = 2 * prec_0 * rec_0 / (prec_0 + rec_0) if (prec_0 + rec_0) > 0 else 0
    
    return (f1_0 + f1_1) / 2
```

## Common pitfalls

- Few-shot prompting performance heavily depends on example selection, which can mislead the model or fail to instill deeper reasoning.
- CoT prompting may violate strict output instructions by appending explanations, requiring post-processing to extract the binary label for fair comparison.
- Performance is evaluated across 18 distinct chunks rather than a single aggregated score, so reporting must account for chunk-level variance.

## Evidence (verbatim from paper)

> We also see a nominal increase in the macro-F1 score in the other remaining chunks. ... Note that, the best-performing baseline, i.e. ConvBERT (as reported in*(Wessel et al., [2023])*) achieves an average Macro-F1 score of $0.7110$. We are achieving performance on par with the prompt-based CoT settings when compared to the ConvBERT model...

## Citation

```bibtex
@misc{sar2025navigating,
  title={Navigating Nuance: In Quest for Political Truth},
  author={Sar et al. (2025)},
  year={2025},
  note={arXiv:2501.00782}
}
```

- arXiv: 2501.00782