Vigil Cognitive Bias Eval

---
name: vigil-cognitive-bias-eval
description: Evaluates a real-time browser extension's ability to detect and mitigate cognitive bias triggers in online text. It probes span-level identification of persuasive rhetoric and bias patterns, alongside system latency and mitigation quality. Use when the user wants to benchmark on SemEval-2020 Task 11, Moralization Corpus, or asks about evaluating this task. Reports micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.03261
  bibtex_key: kang2026vigil
  confidence: high
---

# vigil-cognitive-bias-eval

> VIGIL: An Extensible System for Real-Time Detection and Mitigation of Cognitive Bias Triggers — Kang et al. (2026) (arXiv:2604.03261, 2026)

## What this evaluates

Evaluates a real-time browser extension's ability to detect and mitigate cognitive bias triggers in online text. It probes span-level identification of persuasive rhetoric and bias patterns, alongside system latency and mitigation quality.

## Datasets

- **SemEval-2020 Task 11** — total ?; splits: test (-1)
- **Moralization Corpus** — total ?; splits: test (-1)

## Metrics

- `micro-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, weighted by the number of true instances for each class.
- `macro-F1` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated independently for each class and then averaged unweighted.
- `median latency` — range: seconds
  - Median time in seconds to process a single text span through the detection pipeline.

## Input / output format

**Input**: Text spans from online content (e.g., Twitter/X posts, news articles) presented in real-time as the user scrolls.

**Output**: Span-level detection labels indicating the presence of specific cognitive bias triggers, followed by optional LLM-generated neutralized/reformulated text.

## Scoring recipe

```python
def compute_f1(predictions, gold, average='micro'):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    if average == 'micro':
        return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    else:
        return (precision + recall) / 2
```

## Common pitfalls

- The SemEval-2020 evaluation strictly follows the protocol from Sprenkamp et al. [10], not the original SemEval guidelines.
- Latency measurements are highly backend-dependent (regex: 0.03 ms, WebGPU: 3.4 s, cloud: 3.9 s), making cross-system comparisons invalid without specifying the inference tier.
- The system deliberately optimizes for precision over recall on the SemEval benchmark, which may artificially suppress the reported F1 score.

## Evidence (verbatim from paper)

> On SemEval-2020 Task 11 [3] (using the protocol from Sprenkamp et al. [10]), the production prompt achieves a very competitive micro-F1 = 0.533  with precision  $= 0.626$ , deliberately favoring precision. The moralization plugin achieves macro-F1 = 0.789 on the Moralization Corpus [1], competitive with the corpus authors' best (0.772).

## Citation

```bibtex
@misc{kang2026vigil,
  title={VIGIL: An Extensible System for Real-Time Detection and Mitigation of Cognitive Bias Triggers},
  author={Kang et al. (2026)},
  year={2026},
  note={arXiv:2604.03261}
}
```

- arXiv: 2604.03261