Polar Detection Eval

---
name: polar-detection-eval
description: This benchmark evaluates a model's ability to detect online polarization in social media text by classifying statements as polarized or non-polarized. It specifically probes the model's capacity to produce interpretable, structured reasoning alongside binary predictions while handling class imbalance and reducing false negatives. Use when the user wants to benchmark on POLAR @ SemEval-2026, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11121
  bibtex_key: gupta2026bitspilani
  confidence: high
---

# polar-detection-eval

> BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detection — Gupta et al. (2026) (arXiv:2604.11121, 2026)

## What this evaluates

This benchmark evaluates a model's ability to detect online polarization in social media text by classifying statements as polarized or non-polarized. It specifically probes the model's capacity to produce interpretable, structured reasoning alongside binary predictions while handling class imbalance and reducing false negatives.

## Datasets

- **POLAR @ SemEval-2026** — total 4834; splits: train (3222), dev (160), test (1452)

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across the polarized and non-polarized classes. Calculated as the unweighted mean of the F1 scores for each class, where F1 = 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Raw input text, optionally with referenced target and claim type. For inference, formatted as 'Input: {text}\nReasoning:'.

**Output**: A structured completion containing fields for target, claim type, manifestations (present/absent checklist), decision basis, and a final binary label (0 or 1). The label is extracted via regex from the generated text.

## Scoring recipe

```python
def compute_macro_f1(predictions, golds):
    tp = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, golds) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, golds) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, golds) if p == 0 and g == 0)
    prec_pos = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec_pos = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1_pos = 2 * prec_pos * rec_pos / (prec_pos + rec_pos) if (prec_pos + rec_pos) > 0 else 0.0
    prec_neg = tn / (tn + fn) if (tn + fn) > 0 else 0.0
    rec_neg = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    f1_neg = 2 * prec_neg * rec_neg / (prec_neg + rec_neg) if (prec_neg + rec_neg) > 0 else 0.0
    return (f1_pos + f1_neg) / 2.0
```

## Common pitfalls

- Regex-based label extraction can fail if the model deviates from the strict structured template, leading to dropped predictions.
- The dataset is imbalanced (~36.5% polarized), making false negatives a critical failure mode that standard SFT struggles with.
- Evaluation focuses on the English subset; multilingual performance is not reported in this specific run.

## Evidence (verbatim from paper)

> We establish a simple fine-tuning baseline by splitting the data into train/validation/test (80/10/10), fine-tuning a classifier with LoRA adapters, and evaluating with macro precision, recall, and F1.

## Citation

```bibtex
@misc{gupta2026bitspilani,
  title={BITS Pilani at SemEval-2026 Task 9: Structured Supervised Fine-Tuning with DPO Refinement for Polarization Detection},
  author={Gupta et al. (2026)},
  year={2026},
  note={arXiv:2604.11121}
}
```

- arXiv: 2604.11121