Basqueglue Eval

---
name: basqueglue-eval
description: This benchmark evaluates Basque language models across classical NLP tasks including topic classification, stance detection, coreference detection, and natural language inference. It measures both task-specific performance and overall linguistic competence in a low-resource agglutinative language setting. Use when the user wants to benchmark on BasqueGLUE, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.20266
  bibtex_key: etxaniz2024latxa
  confidence: high
---

# basqueglue-eval

> Latxa: An Open Language Model and Evaluation Suite for Basque — Etxaniz et al. (2024) (arXiv:2403.20266, 2024)

## What this evaluates

This benchmark evaluates Basque language models across classical NLP tasks including topic classification, stance detection, coreference detection, and natural language inference. It measures both task-specific performance and overall linguistic competence in a low-resource agglutinative language setting.

## Datasets

- **BasqueGLUE** — total ?; splits: test (-1); repo https://github.com/hitz-zentroa/atxa

## Metrics

- `F1` — range: [0, 1]
  - Standard F1 score for classification tasks. For VaxxStance, it is computed as a macro-average over the IN FAVOUR and AGAINST categories only.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances.
- `Avg` **(primary)** — range: [0, 1]
  - Arithmetic mean of the task-specific scores (F1 or Accuracy) across all BasqueGLUE tasks.

## Input / output format

**Input**: Text prompts with task instructions (few-shot or zero-shot for decoder models) or raw text sequences (for encoder models).

**Output**: Predicted class label or probability distribution over classes.

## Scoring recipe

```python
def compute_avg_score(task_results):
    scores = []
    for task, res in task_results.items():
        if task == 'VaxxStance':
            scores.append(macro_f1(res['golds'], res['preds'], pos=['IN FAVOUR', 'AGAINST']))
        else:
            scores.append(f1_score(res['golds'], res['preds']) if 'F1' in task else accuracy_score(res['golds'], res['preds']))
    return sum(scores) / len(scores)
```

## Common pitfalls

- VaxxStance F1 is macro-averaged only over IN FAVOUR and AGAINST, excluding other categories.
- Evaluation settings vary by model type (few-shot for decoders vs. fine-tuned for encoders), making direct performance comparisons across architectures tricky.
- The average score aggregates heterogeneous tasks without normalization, which can skew the overall benchmark score.

## Evidence (verbatim from paper)

> Table 6: BasqueGLUE results by task. *VaxxStance is measured in terms of macro-average F1-score of the categories IN FAVOUR and AGAINST. †BERTeus and ElhBERTeu are fine-tuned encoders.

## Citation

```bibtex
@misc{etxaniz2024latxa,
  title={Latxa: An Open Language Model and Evaluation Suite for Basque},
  author={Etxaniz et al. (2024)},
  year={2024},
  note={arXiv:2403.20266}
}
```

- arXiv: 2403.20266