Vietnamese Abusive Span Detection Eval

---
name: vietnamese-abusive-span-detection-eval
description: Identifies and categorizes abusive content spans within long-form Vietnamese narrative texts. It probes a model's ability to perform sequence labeling for both span detection and fine-grained abuse classification across six distinct categories. Use when the user wants to benchmark on Vietnamese Narrative Abusive Span Dataset, or asks about evaluating this task. Reports Strict F-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.07831
  bibtex_key: nguyen2023abusive
  confidence: high
---

# vietnamese-abusive-span-detection-eval

> Abusive Span Detection for Vietnamese Narrative Texts — Nguyen et al. (2023) (arXiv:2312.07831, 2023)

## What this evaluates

Identifies and categorizes abusive content spans within long-form Vietnamese narrative texts. It probes a model's ability to perform sequence labeling for both span detection and fine-grained abuse classification across six distinct categories.

## Datasets

- **Vietnamese Narrative Abusive Span Dataset** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `Strict F-score` **(primary)** — range: percent
  - Precision = Total exact matching spans / Total spans predicted by the system. Recall = Total exact matching spans / Total gold spans. F-score = 2 * Precision * Recall / (Precision + Recall). Requires exact start and end boundary matching for each span.
- `Relaxed F-score` — range: percent
  - Precision = Total exact matching characters / Total characters predicted by the system. Recall = Total exact matching characters / Total gold characters. F-score = 2 * Precision * Recall / (Precision + Recall). Assesses only the positive class with character-level overlap.

## Input / output format

**Input**: Vietnamese narrative text passages, processed via frozen pre-trained language model (PhoBERT or XLM-RoBERTa) hidden states projected to dimension 128, with the first sub-word token from each pre-trained model fed into a BiLSTM.

**Output**: Span boundaries (start and end indices) for abusive content, optionally annotated with one of six abuse categories: Cultural Identity, Financial/Economic, Mental/Psychological, Physical, Sexual, or Verbal/Emotional.

## Scoring recipe

```python
def compute_fscore(pred_spans, gold_spans, mode='strict'):
    if mode == 'strict':
        tp = sum(1 for p in pred_spans if p in gold_spans)
        p = tp / len(pred_spans) if pred_spans else 0
        r = tp / len(gold_spans) if gold_spans else 0
    else:
        pred_chars = sum(len(s) for s in pred_spans)
        gold_chars = sum(len(s) for s in gold_spans)
        matched_chars = sum(len(s) for s in pred_spans if s in gold_spans)
        p = matched_chars / pred_chars if pred_chars else 0
        r = matched_chars / gold_chars if gold_chars else 0
    f = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return f * 100
```

## Common pitfalls

- Strict evaluation demands exact boundary matching, yielding significantly lower scores than relaxed evaluation which uses character-level overlap.
- Class imbalance heavily skews results; categories like Sexual and Cultural/Identity comprise <3% of data, leading to 0% F-scores regardless of model capability.
- Relaxed evaluation only assesses the positive class, which intentionally depresses metrics to leave room for improvement.

## Evidence (verbatim from paper)

> In the context of the relaxed evaluation, which is inspired by the approach in [14], we proposed to exclusively assess the positive class. This choice deliberately yields lower metrics and offers ample room for future enhancements and refinements. So, the F-score for the strict evaluation is computed using the following formulations.

$$
\text {P r e c i s i o n} = \frac {\text {T o t a l e x a c t m a t c h i n g s p a n s}}{\text {T o t a l s p a n s p r e d i c t e d b y t h e s y s t e m}} \tag {1}
$$

$$
\text {R e c a l l} = \frac {\text {T o t a l e x a c t m a t c h i n g s p a n s}}{\text {T o t a l g o l d s p a n s}} \tag {2}
$$

$$
F - s c o r e = \frac {2 \cdot \text {P r e c i s i o n} \cdot \text {R e c a l l}}{\text {P r e c i s i o n} + \text {R e c a l l}} \tag {3}
$$

## Citation

```bibtex
@misc{nguyen2023abusive,
  title={Abusive Span Detection for Vietnamese Narrative Texts},
  author={Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2312.07831}
}
```

- arXiv: 2312.07831