Propaganda Detection Eval

---
name: propaganda-detection-eval
description: Detects propaganda techniques in news articles through a two-stage pipeline: identifying text spans containing propaganda (Span Identification) and classifying the specific rhetorical technique used within those spans (Technique Classification). Use when the user wants to benchmark on SemEval-2020 Task 11 Propaganda Detection, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.02696
  bibtex_key: [REDACTED]
  confidence: high
---

# propaganda-detection-eval

> SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles — Da San Martino et al. (2020) (arXiv:2009.02696, 2020)

## What this evaluates

Detects propaganda techniques in news articles through a two-stage pipeline: identifying text spans containing propaganda (Span Identification) and classifying the specific rhetorical technique used within those spans (Technique Classification).

## Datasets

- **SemEval-2020 Task 11 Propaganda Detection** — total ?; splits: test (-1), development (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. For Span Identification, computed at the character level over identified spans. For Technique Classification, macro-averaged across 14 propaganda technique classes.

## Input / output format

**Input**: News article text with character-level annotations for propaganda spans and their corresponding technique labels.

**Output**: For Span Identification: a list of character spans flagged as propaganda. For Technique Classification: a classification label from 14 predefined propaganda techniques for each identified span.

## Scoring recipe

```python
def compute_f1(pred_spans, gold_spans, pred_labels, gold_labels):
    # Span Identification (character-level)
    tp = len(set(pred_spans) & set(gold_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_si = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    # Technique Classification (macro-averaged)
    f1_tc = macro_f1(pred_labels, gold_labels)
    return f1_si, f1_tc
```

## Common pitfalls

- Overfitting to the development set can cause significant performance drops on the test set, as observed for teams syrapropa and PALI.
- Combining systems via union increases recall but decreases precision, while intersection does the opposite; majority voting offers a balance.
- Rare propaganda techniques (e.g., Straw man, Bandwagon) are hard to classify, skewing overall macro-F1 performance.

## Evidence (verbatim from paper)

> Figure 6: F1 performance for the technique classification subtask when combining up to top-20 systems with majority voting. The plots show the overall performance (top left) as well as for each of the 14 classes.

## Citation

```bibtex
@misc{dasanmartino2020semeval,
  title={SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles},
  author={Da San Martino et al. (2020)},
  year={2020},
  note={arXiv:2009.02696}
}
```

- arXiv: 2009.02696