Swsr Eval

---
name: swsr-eval
description: Probes the capability of NLP models to detect online sexism in Chinese microblogging comments. It evaluates performance across three hierarchical classification tasks: binary sexism identification, fine-grained category classification, and target type classification. Use when the user wants to benchmark on SWSR, or asks about evaluating this task. Reports macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.03070
  bibtex_key: jiang2021swsr
  confidence: high
---

# swsr-eval

> SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection — Jiang et al. (2021) (arXiv:2108.03070, 2021)

## What this evaluates

Probes the capability of NLP models to detect online sexism in Chinese microblogging comments. It evaluates performance across three hierarchical classification tasks: binary sexism identification, fine-grained category classification, and target type classification.

## Datasets

- **SWSR** — total 8969; splits: train (-1), test (-1)

## Metrics

- `macro F1` **(primary)** — range: [0, 1]
  - The unweighted mean of the F1 scores computed for each class independently. F1 for a class is 2 * (precision * recall) / (precision + recall), where precision and recall are calculated per class.
- `accuracy` — range: [0, 1]
  - The proportion of correctly classified instances out of the total number of instances.
- `weighted F1` — range: [0, 1]
  - The mean of F1 scores weighted by the number of true instances for each class, accounting for class imbalance.

## Input / output format

**Input**: Raw Chinese text of online comments or weibo posts.

**Output**: Discrete class label corresponding to the task: 'sexist' or 'non-sexist' (binary); 'SA', 'SCB', 'MA', 'SO', or 'non-sexist' (category); or 'generic', 'individual', or 'non-sexist' (target).

## Scoring recipe

```python
def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = sorted(set(golds) | set(preds))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return macro_f1, acc
```

## Common pitfalls

- The dataset is highly imbalanced across fine-grained classes, making accuracy and weighted F1 potentially misleading; macro F1 is explicitly reported to mitigate this bias.
- Chinese text lacks explicit word boundaries, so character-level features or models are required for competitive performance, unlike standard word-level n-gram baselines.
- Lexicon integration uses TF-IDF on raw word counts concatenated with embeddings, which may not align well with contextual transformer representations and yields only marginal gains.

## Evidence (verbatim from paper)

> We report global macro F1 and accuracy scores for the three tasks, as well as F1 scores specific to each class for experimental step 1 and weighted F1 scores for steps 2 and 3.

## Citation

```bibtex
@misc{jiang2021swsr,
  title={SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection},
  author={Jiang et al. (2021)},
  year={2021},
  note={arXiv:2108.03070}
}
```

- arXiv: 2108.03070