Filipino Text Benchmarks Eval

---
name: filipino-text-benchmarks-eval
description: Evaluates text classification capability in low-resource settings for the Filipino language, specifically measuring model robustness and performance degradation as training data size is systematically reduced. Use when the user wants to benchmark on Hate Speech, Dengue, or asks about evaluating this task. Reports accuracy, hamming loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.02068
  bibtex_key: cruz2020establishing
  confidence: high
---

# filipino-text-benchmarks-eval

> Establishing Baselines for Text Classification in Low-Resource Languages — Cruz et al. (2020) (arXiv:2005.02068, 2020)

## What this evaluates

Evaluates text classification capability in low-resource settings for the Filipino language, specifically measuring model robustness and performance degradation as training data size is systematically reduced.

## Datasets

- **Hate Speech** — total ?; splits: train (-1), test (-1); repo https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
- **Dengue** — total ?; splits: train (-1), test (-1); repo https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: fraction of correctly predicted labels out of total test instances.
- `hamming loss` **(primary)** — range: [0, 1]
  - Fraction of incorrect labels in a multilabel setting; lower is better.
- `% degradation` — range: percent
  - Relative performance loss compared to the full-dataset baseline: (Baseline_Metric - Reduced_Data_Metric) / Baseline_Metric * 100.

## Input / output format

**Input**: Raw Filipino text strings.

**Output**: Predicted class label(s) for each text instance.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_hamming_loss(preds, gold):
    total = len(preds) * len(preds[0])
    incorrect = sum(1 for p, g in zip(preds, gold) if set(p) != set(g))
    return incorrect / total

def compute_degradation(baseline_val, reduced_val):
    return ((baseline_val - reduced_val) / baseline_val) * 100
```

## Common pitfalls

- Failing to keep the validation and test sets constant across different training data sizes, which invalidates the degradation comparison.
- Computing degradation as an absolute difference rather than the specified relative percentage.
- Not repeating the finetuning process 5 times (k=5 cross-validation) to account for variance in low-data settings.

## Evidence (verbatim from paper)

> Degradation is computed by subtracting the full 10k accuracy with the accuracy when trained with less samples, then dividing by the full 10k accuracy. We log the test loss and accuracy for each reduction, then compute for the differences against the loss and accuracy when trained with the full dataset. After which, we calculate the degradation percentage, which is the amount of performance loss the model experiences relative to its accuracy when trained with the full dataset.

## Citation

```bibtex
@misc{cruz2020establishing,
  title={Establishing Baselines for Text Classification in Low-Resource Languages},
  author={Cruz et al. (2020)},
  year={2020},
  note={arXiv:2005.02068}
}
```

- arXiv: 2005.02068