News Bias Detection Eval

---
name: news-bias-detection-eval
description: Evaluates transformer models' ability to classify news articles as biased or unbiased, comparing standard fine-tuning against domain-adapted training. It further probes model decision-making by analyzing word-level SHAP attribution magnitudes and lexical feature importance across true and false predictions. Use when the user wants to benchmark on BABE, or asks about evaluating this task. Reports Binary F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.23835
  bibtex_key: ghosh2025explainingnewsbias
  confidence: high
---

# news-bias-detection-eval

> Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms — Ghosh (2025) (arXiv:2512.23835, 2025)

## What this evaluates

Evaluates transformer models' ability to classify news articles as biased or unbiased, comparing standard fine-tuning against domain-adapted training. It further probes model decision-making by analyzing word-level SHAP attribution magnitudes and lexical feature importance across true and false predictions.

## Datasets

- **BABE** — total ?; splits: test (-1)

## Metrics

- `Binary F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall).
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive instances to all instances predicted as positive.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive instances to all actual positive instances.
- `Accuracy` — range: [0, 1]
  - Ratio of correct predictions to total predictions.
- `False Positive Rate` — range: [0, 1]
  - Ratio of false positives to all actual negative instances.
- `Macro F1` — range: [0, 1]
  - Unweighted mean of F1 scores for each class.
- `Weighted F1` — range: [0, 1]
  - Mean of F1 scores weighted by class support.

## Input / output format

**Input**: News article text (paragraphs or full articles) to be classified for the presence of political/social bias.

**Output**: Binary label: biased or unbiased (not biased).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    tp = sum((p == 1) & (g == 1) for p, g in zip(preds, golds))
    fp = sum((p == 1) & (g == 0) for p, g in zip(preds, golds))
    fn = sum((p == 0) & (g == 1) for p, g in zip(preds, golds))
    tn = sum((p == 0) & (g == 0) for p, g in zip(preds, golds))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'binary_f1': f1, 'fpr': fpr}
```

## Common pitfalls

- The paper evaluates two distinct models on the same test set; results must be compared fairly without mixing training data or evaluation splits.
- SHAP attribution magnitudes are analyzed conditionally on prediction correctness (TP vs FP), not just globally; misinterpreting this can lead to wrong conclusions about model alignment.
- Statistical significance is assessed via McNemar's test on the contingency table of errors, not via standard t-tests on metric scores.

## Evidence (verbatim from paper)

> Table [1] summarizes the performance metrics for both models. While bias-detector achieved higher accuracy and F1, DA-RoBERTa-BABE-FT showed better precision. The significant difference in false positive rates (15.6% vs. 5.7%) suggests that architectural differences and training approaches substantially affect precision in bias detection tasks.

## Citation

```bibtex
@misc{ghosh2025explainingnewsbias,
  title={Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms},
  author={Ghosh (2025)},
  year={2025},
  note={arXiv:2512.23835}
}
```

- arXiv: 2512.23835