Slovene Superglue Eval

---
name: slovene-superglue-eval
description: Evaluates monolingual, cross-lingual, and multilingual NLP models on a human- and machine-translated Slovene version of the SuperGLUE benchmark. It probes how well models handle morphological and grammatical challenges in low-resource language processing, and compares translation quality impacts on downstream task performance. Use when the user wants to benchmark on Slovene SuperGLUE, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.04994
  bibtex_key: zagar2022slovenesuperglue
  confidence: high
---

# slovene-superglue-eval

> Slovene SuperGLUE Benchmark: Translation and Evaluation — Žagar et al. (2022) (arXiv:2202.04994, 2022)

## What this evaluates

Evaluates monolingual, cross-lingual, and multilingual NLP models on a human- and machine-translated Slovene version of the SuperGLUE benchmark. It probes how well models handle morphological and grammatical challenges in low-resource language processing, and compares translation quality impacts on downstream task performance.

## Datasets

- **Slovene SuperGLUE** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Avg` **(primary)** — range: percent
  - Average of task-specific scores across the evaluated tasks. Task scores are Accuracy (Acc.), F1/Accuracy (for CB), or F1a/EM (for MultiRC). Note that the set of tasks averaged differs by experiment (6 tasks in Table 2, 5 tasks excluding WSC in Table 5).
- `Accuracy (Acc.)` — range: percent
  - Percentage of correctly predicted class labels for tasks like BoolQ, COPA, RTE, and WSC.
- `F1/Accuracy` — range: percent
  - Reported as F1/Accuracy for the CB task, evaluating both macro-F1 and accuracy.
- `F1a/EM` — range: percent
  - Reported for MultiRC, evaluating average F1 and exact match for multiple-choice answers.

## Input / output format

**Input**: Slovene text instances corresponding to SuperGLUE tasks (BoolQ, CB, COPA, MultiRC, RTE, WSC), including premises, hypotheses, questions, and context as appropriate for each task.

**Output**: Predicted class labels or answer spans for each instance.

## Scoring recipe

```python
def compute_avg(predictions, gold):
    task_scores = {}
    for task in ['BoolQ', 'CB', 'COPA', 'MultiRC', 'RTE', 'WSC']:
        if task == 'CB':
            task_scores[task] = f1_accuracy(predictions[task], gold[task])
        elif task == 'MultiRC':
            task_scores[task] = f1a_em(predictions[task], gold[task])
        else:
            task_scores[task] = accuracy(predictions[task], gold[task])
    return sum(task_scores.values()) / len(task_scores)
```

## Common pitfalls

- The average score (Avg) is computed differently across tables: Table 2 averages over 6 tasks, while Table 5 excludes WSC and averages over 5 tasks.
- Small training set sizes for BoolQ and MultiRC cause models to default to predicting the most frequent class, making performance metrics unreliable for those tasks.
- WSC requires human translation and is excluded from machine vs. human translation comparison experiments.

## Evidence (verbatim from paper)

> Considering the Avg scores in Table 2, the monolingual SloBERTa is the best performing Slovene model. On average, all Slovene BERT models perform better than the Most Frequent baseline.

## Citation

```bibtex
@misc{zagar2022slovenesuperglue,
  title={Slovene SuperGLUE Benchmark: Translation and Evaluation},
  author={Žagar et al. (2022)},
  year={2022},
  note={arXiv:2202.04994}
}
```

- arXiv: 2202.04994