Safe Speech Eval

---
name: safe-speech-eval
description: Evaluates the capability of classifiers to detect sexist, abusive, offensive, and hate speech in conversational text across multiple granularity levels and established benchmarks. It probes fine-grained toxicity detection, cross-dataset generalization, and performance against strong supervised and LLM baselines. Use when the user wants to benchmark on EDOS (SemEval 2023), OffensEval 2019, AbusEval, HatEval, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.06534
  bibtex_key: tan2025safespeech
  confidence: medium
---

# safe-speech-eval

> SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversations — Tan et al. (2025) (arXiv:2503.06534, 2025)

## What this evaluates

Evaluates the capability of classifiers to detect sexist, abusive, offensive, and hate speech in conversational text across multiple granularity levels and established benchmarks. It probes fine-grained toxicity detection, cross-dataset generalization, and performance against strong supervised and LLM baselines.

## Datasets

- **EDOS (SemEval 2023)** — total ?; splits: test (-1)
- **OffensEval 2019** — total ?; splits: test (-1)
- **AbusEval** — total ?; splits: test (-1)
- **HatEval** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as absolute scores or percentage improvements over baselines.

## Input / output format

**Input**: Textual messages or dialogue turns annotated for toxicity, sexism, hate speech, or abuse.

**Output**: Predicted labels corresponding to the evaluation subtask (binary detection, category-level classification, or fine-grained vector-level classification).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g and p == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
```

## Common pitfalls

- The paper reports F1 scores without specifying whether macro, micro, or weighted averaging is used, which is critical for multi-class and vector-level tasks like Subtask B and C.
- Out-of-domain evaluation is performed (training on EDOS, testing on OffensEval/AbusEval/HatEval), which may not reflect true generalization to unseen conversational contexts.
- Granularity levels differ across subtasks (binary vs. category vs. vector), making direct cross-subtask performance comparisons misleading without normalization.

## Evidence (verbatim from paper)

> The results demonstrate that M7-FE achieves top performance across all subtasks, improving over the winning system Zhou ([2023])* by 1% F1 on Subtask A and 4% F1 on Subtask C. On Subtask B, M7-FE matches the best reported performance while outperforming publicly available baselines such as ISEGURA/roberta-base_edos_b by 14% F1.

## Citation

```bibtex
@misc{tan2025safespeech,
  title={SafeSpeech: A Comprehensive and Interactive Tool for Analysing Sexist and Abusive Language in Conversations},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2503.06534}
}
```

- arXiv: 2503.06534