Ltzglue Eval

---
name: ltzglue-eval
description: Evaluates Luxembourgish (LTZ) language understanding across eight diverse NLU tasks, including classification, sequence labeling, and textual entailment. It probes encoder models and prompted LLMs on their ability to handle low-resource language nuances, structural complexity, and label sensitivity. Use when the user wants to benchmark on ltzGLUE, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17976
  bibtex_key: plum2026ltzglue
  confidence: high
---

# ltzglue-eval

> ltzGLUE: Luxembourgish General Language Understanding Evaluation — Plum et al. (2026) (arXiv:2604.17976, 2026)

## What this evaluates

Evaluates Luxembourgish (LTZ) language understanding across eight diverse NLU tasks, including classification, sequence labeling, and textual entailment. It probes encoder models and prompted LLMs on their ability to handle low-resource language nuances, structural complexity, and label sensitivity.

## Datasets

- **ltzGLUE** — total ?; splits: test (-1); repo https://github.com/plumaj/ltzGLUE

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Unweighted mean of per-class F1 scores. Computed as 2 * (precision * recall) / (precision + recall) for each class, then averaged across all classes.

## Input / output format

**Input**: Luxembourgish text inputs for classification, sequence labeling, or entailment tasks.

**Output**: Predicted class labels or token-level tags corresponding to the specific task.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    classes = sorted(set(predictions) | set(gold))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Prompted LLMs may produce malformed outputs or incorrect numbers of predictions, which are discarded prior to evaluation.
- Encoder results are averaged over three runs with standard deviations reported, while LLMs are evaluated only once, making stability comparisons difficult.
- High class imbalance in HA and SA requires class-balanced loss during fine-tuning, which affects training but not the test metric itself.

## Evidence (verbatim from paper)

> Table 6 shows F1 scores for all models across all tasks (see Appendix 7.9 for full results). Prompted LLMs were evaluated once; we report macro-F1 only.

## Citation

```bibtex
@misc{plum2026ltzglue,
  title={ltzGLUE: Luxembourgish General Language Understanding Evaluation},
  author={Plum et al. (2026)},
  year={2026},
  note={arXiv:2604.17976}
}
```

- arXiv: 2604.17976