Social Media Bias Eval

---
name: social-media-bias-eval
description: This benchmark evaluates the ability of models to automatically detect multiple dimensions of media bias (e.g., hate speech, racial, gender, political, linguistic, and text-level context bias) in social media posts across different topic domains. It probes a model's robustness to domain shift and severe class imbalance in multi-label bias identification tasks. Use when the user wants to benchmark on Social Media Bias Dataset (YouTube & Reddit), or asks about evaluating this task. Reports weighted average F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.15406
  bibtex_key: liu2024intertwined
  confidence: high
---

# social-media-bias-eval

> Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensions — Liu et al. (2024) (arXiv:2408.15406, 2024)

## What this evaluates

This benchmark evaluates the ability of models to automatically detect multiple dimensions of media bias (e.g., hate speech, racial, gender, political, linguistic, and text-level context bias) in social media posts across different topic domains. It probes a model's robustness to domain shift and severe class imbalance in multi-label bias identification tasks.

## Datasets

- **Social Media Bias Dataset (YouTube & Reddit)** — total ?; splits: test (-1)

## Metrics

- `weighted average F1 score` **(primary)** — range: [0, 1]
  - The F1 score is computed for each class (positive/negative for each bias dimension) and then averaged, weighting each class by its support (number of true instances). This accounts for the severe class imbalance where less than 20% of annotations are positive.

## Input / output format

**Input**: Raw text of social media posts from YouTube and Reddit, categorized into five topic domains (politics, sports, healthcare, job & education, entertainment).

**Output**: Binary labels (positive/negative) for each of the six bias dimensions: Gender Bias, Racial Bias, Hate Speech, Linguistic Bias, Text-level Context Bias, and Political Bias.

## Scoring recipe

```python
def compute_weighted_f1(y_true, y_pred):
    f1_scores = []
    weights = []
    for label in [0, 1]:
        tp = np.sum((y_true == label) & (y_pred == label))
        fp = np.sum((y_true != label) & (y_pred == label))
        fn = np.sum((y_true == label) & (y_pred != label))
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
        weights.append(np.sum(y_true == label))
    return np.average(f1_scores, weights=weights)
```

## Common pitfalls

- Severe class imbalance (<20% positive annotations) makes accuracy misleading; weighted F1 is required.
- Models trained on external datasets (e.g., MBIB) suffer significant performance drops due to domain shift, making cross-dataset evaluation unreliable without fine-tuning.
- Nuanced definitions of political, linguistic, and text-level context biases lead to inherently lower prediction performance and lower inter-rater agreement.

## Evidence (verbatim from paper)

> To account for class imbalance, we use weighted average F1 score as our evaluation metrics for all our automated annotations following practices in prior works [13], [33]. For the models trained on MBIB datasets, we adopt a random 5-fold train-validation split with a 75% data used for training and 25% data used for validation, where the best-performing model in validation set is used for evaluation on our collected social media posts.

## Citation

```bibtex
@misc{liu2024intertwined,
  title={Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensions},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2408.15406}
}
```

- arXiv: 2408.15406