Vihos Eval

---
name: vihos-eval
description: Evaluates a model's ability to detect hate and offensive text spans within Vietnamese social media comments. It probes sequence tagging capabilities, specifically requiring precise boundary identification of offensive content in noisy, informal text. Use when the user wants to benchmark on ViHOS, or asks about evaluating this task. Reports macro-average F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.10186
  bibtex_key: hoang2023vihos
  confidence: high
---

# vihos-eval

> ViHOS: Hate Speech Spans Detection for Vietnamese — Hoang et al. (2023) (arXiv:2301.10186, 2023)

## What this evaluates

Evaluates a model's ability to detect hate and offensive text spans within Vietnamese social media comments. It probes sequence tagging capabilities, specifically requiring precise boundary identification of offensive content in noisy, informal text.

## Datasets

- **ViHOS** — total 11056; splits: test (-1), dev (-1); repo https://github.com/phusroyal/ViHOS

## Metrics

- `macro-average F1-score` **(primary)** — range: [0, 1]
  - Computed as the arithmetic mean of F1 scores across all gold-predicted span pairs. Precision, Recall, and F1 are calculated at the span level, then averaged across the dataset.

## Input / output format

**Input**: Vietnamese social media comments (text), tokenized at either syllable-level or word-level.

**Output**: IOB (Inside-Outside-Beginning) character-level tags indicating the start and end boundaries of hate/offensive spans.

## Scoring recipe

```python
def compute_span_f1(gold_spans, pred_spans):
    tp = len(set(gold_spans) & set(pred_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0

macro_f1 = sum(compute_span_f1(g, p) for g, p in zip(gold_spans_list, pred_spans_list)) / len(gold_spans_list)
```

## Common pitfalls

- Tokenization granularity (syllable vs. word) drastically changes performance depending on the model architecture.
- Censored text (e.g., asterisks like 'b**i') and informal social media formatting often cause incorrect span boundary predictions.
- Context-dependent metaphors, idioms, and allusions lead to frequent false negatives even for strong pre-trained models.

## Evidence (verbatim from paper)

> The macro-average F1-score (F1) is used to evaluate our models. For each pair of gold-predicted spans, we compute F1 and then calculate the arithmetic mean of F1 for each of these cases. It should be noted that the final F1-score, Accuracy, and Precision reported are an average of more than ten runs with various random seeds.

## Citation

```bibtex
@misc{hoang2023vihos,
  title={ViHOS: Hate Speech Spans Detection for Vietnamese},
  author={Hoang et al. (2023)},
  year={2023},
  note={arXiv:2301.10186}
}
```

- arXiv: 2301.10186