Vlue Eval

---
name: vlue-eval
description: Evaluates Vietnamese natural language understanding across five diverse tasks including machine reading comprehension, natural language inference, emotion recognition, hate speech detection, and part-of-speech tagging. It assesses a model's ability to comprehend text, reason over sentence pairs, classify emotions and hate speech, and perform syntactic analysis in Vietnamese. Use when the user wants to benchmark on UIT-ViQuAD 2.0, ViNLI, VSMEC, ViHOS, NIIVTB POS, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.15882
  bibtex_key: do2024vlue
  confidence: high
---

# vlue-eval

> VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding — Do et al. (2024) (arXiv:2403.15882, 2024)

## What this evaluates

Evaluates Vietnamese natural language understanding across five diverse tasks including machine reading comprehension, natural language inference, emotion recognition, hate speech detection, and part-of-speech tagging. It assesses a model's ability to comprehend text, reason over sentence pairs, classify emotions and hate speech, and perform syntactic analysis in Vietnamese.

## Datasets

- **UIT-ViQuAD 2.0** — total 35990; splits: train (28457), dev (3821), test (3712)
- **ViNLI** — total 30376; splits: train (24376), dev (3009), test (2991)
- **VSMEC** — total 6927; splits: train (5548), dev (686), test (693)
- **ViHOS** — total 11214; splits: train (8974), dev (1112), test (1128)
- **NIIVTB POS** — total 20588; splits: train (18588), dev (1000), test (1000)

## Metrics

- `Exact Match (EM)` — range: [0, 1]
  - 1 if the predicted answer exactly matches the gold answer span, else 0.
- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall computed over token or character overlap between predicted and gold spans/labels.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total number of instances.
- `Macro-F1` — range: [0, 1]
  - Unweighted mean of F1 scores computed independently for each class, then averaged.

## Input / output format

**Input**: Varies by task: (context, question) for MRC; (premise, hypothesis) for NLI; (comment text) for emotion/hate speech; (sentence tokens) for POS tagging.

**Output**: Varies by task: extracted span or empty string for MRC; class label (entailment/neutral/contradiction/other) for NLI; emotion label(s) for emotion recognition; hate/offensive span(s) or none for hate speech; POS tag sequence for POS tagging.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    em = sum(1.0 if p == g else 0.0 for p, g in zip(preds, golds)) / len(golds)
    f1s = []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p.split()), set(g.split())
        if not p_set and not g_set: f1s.append(1.0)
        elif not p_set or not g_set: f1s.append(0.0)
        else:
            prec = len(p_set & g_set) / len(p_set)
            rec = len(p_set & g_set) / len(g_set)
            f1s.append(2 * prec * rec / (prec + rec) if prec + rec > 0 else 0.0)
    return {'EM': em, 'F1': sum(f1s) / len(f1s)}
```

## Common pitfalls

- Models must predict an empty span for unanswerable questions in UIT-ViQuAD; failing to do so penalizes EM/F1.
- VSMEC is a multi-label classification task, so macro-F1 must be computed per label rather than using standard single-label accuracy.
- ViHOS requires span-level extraction, not just comment-level classification, so evaluation must match predicted spans to gold spans character-by-character.

## Evidence (verbatim from paper)

> The task proposed by this dataset is to extract the answer for a question given a corresponding context. The answer can be empty when models encounter unanswerable questions. Exact Match (EM) and F1-score are used to evaluate the performance of the model.

## Citation

```bibtex
@misc{do2024vlue,
  title={VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding},
  author={Do et al. (2024)},
  year={2024},
  note={arXiv:2403.15882}
}
```

- arXiv: 2403.15882