Vukuzenzele Za Gov Eval

---
name: vukuzenzele-za-gov-eval
description: This benchmark evaluates multilingual sentence alignment and machine translation capabilities across 11 official South African languages using government-themed corpora. Use when the user wants to benchmark on Vuk'uzenzele, ZA-gov-multilingual, or asks about evaluating this task. Reports cosine similarity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.03750
  bibtex_key: lastrucci2023vukuzenzele
  confidence: medium
---

# vukuzenzele-za-gov-eval

> Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora — Lastrucci et al. (2023) (arXiv:2303.03750, 2023)

## What this evaluates

This benchmark evaluates multilingual sentence alignment and machine translation capabilities across 11 official South African languages using government-themed corpora.

## Datasets

- **Vuk'uzenzele** — total ?; splits: train (-1); repo https://github.com/dsfsi/vukuzenzele-nlp
- **ZA-gov-multilingual** — total ?; splits: train (-1); repo https://github.com/dsfsi/gov-za-multilingual

## Metrics

- `cosine similarity` **(primary)** — range: [0, 1]
  - Cosine similarity between 1024-dimensional LASER sentence embeddings. Scores range from 0 to 1, with higher values indicating greater semantic similarity.

## Input / output format

**Input**: Preprocessed source and target sentences in 11 official South African languages.

**Output**: CSV files named 'aligned-{src_lang_code}-{tgt_lang_code}.csv' containing source sentence, target sentence, and cosine similarity score.

## Scoring recipe

```python
# Pseudo-code
src_tokens = nltk_tokenize(preprocess(src_text))
tgt_tokens = nltk_tokenize(preprocess(tgt_text))
src_vecs = LASER.encode(src_tokens)  # 1024-dim
tgt_vecs = LASER.encode(tgt_tokens)  # 1024-dim
scores = cosine_similarity(src_vecs, tgt_vecs)
aligned = [(s, t, sc) for s, t, sc in zip(src_tokens, tgt_tokens, scores) if sc >= 0.65]
```

## Common pitfalls

- Dataset sizes vary significantly across language pairs due to differences in scraping outcomes and source material availability.
- Preprocessing steps differ between corpora (e.g., ZA-Gov requires removing nested numbering, Vuk'uzenzele requires removing bullet points), which can affect tokenization and alignment quality.
- The fixed cosine similarity threshold of 0.65 is arbitrary and may exclude valid translations or include semantically similar but non-equivalent sentences.

## Evidence (verbatim from paper)

> The sentence vectors are compared and a cosine similarity algorithm was performed to produce a score from 0 to 1 on the similarity of the two vectors as described in section 4.2.1. The observations present in each dataset with a score above 0.65, or 65% similarity, are listed in the following tables 2 and 3.

## Citation

```bibtex
@misc{lastrucci2023vukuzenzele,
  title={Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora},
  author={Lastrucci et al. (2023)},
  year={2023},
  note={arXiv:2303.03750}
}
```

- arXiv: 2303.03750