Portulan Extraglue Eval

---
name: portulan-extraglue-eval
description: Evaluates neural models on a Portuguese-language benchmark derived from English GLUE and SuperGLUE tasks, probing capabilities in grammatical acceptability, sentiment, paraphrase detection, semantic similarity, natural language inference, reading comprehension, and causal reasoning. Use when the user wants to benchmark on CoLA, SST-2, MRPC, QQP, STS-B, WiC, MNLI, QNLI, RTE, WNLI, WSC, CB, AXb, AXg, BoolQ, MultiRC, ReCoRD, COPA, or asks about evaluating this task. Reports single-number performance metric.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.05333
  bibtex_key: osorio2024portulanextraglue
  confidence: medium
---

# portulan-extraglue-eval

> PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese — Tomás Freitas Osório et al. (2024) (arXiv:2404.05333, 2024)

## What this evaluates

Evaluates neural models on a Portuguese-language benchmark derived from English GLUE and SuperGLUE tasks, probing capabilities in grammatical acceptability, sentiment, paraphrase detection, semantic similarity, natural language inference, reading comprehension, and causal reasoning.

## Datasets

- **CoLA** — total ?; splits: train (-1), val (-1), test (-1)
- **SST-2** — total ?; splits: train (-1), val (-1), test (-1)
- **MRPC** — total ?; splits: train (-1), val (-1), test (-1)
- **QQP** — total ?; splits: train (-1), val (-1), test (-1)
- **STS-B** — total ?; splits: train (-1), val (-1), test (-1)
- **WiC** — total ?; splits: train (-1), val (-1), test (-1)
- **MNLI** — total ?; splits: train (-1), val (-1), test (-1)
- **QNLI** — total ?; splits: train (-1), val (-1), test (-1)
- **RTE** — total ?; splits: train (-1), val (-1), test (-1)
- **WNLI** — total ?; splits: train (-1), val (-1), test (-1)
- **WSC** — total ?; splits: train (-1), val (-1), test (-1)
- **CB** — total ?; splits: train (-1), val (-1), test (-1)
- **AXb** — total ?; splits: test (-1)
- **AXg** — total ?; splits: test (-1)
- **BoolQ** — total ?; splits: train (-1), val (-1), test (-1)
- **MultiRC** — total ?; splits: train (-1), val (-1), test (-1)
- **ReCoRD** — total ?; splits: train (-1), val (-1), test (-1)
- **COPA** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `single-number performance metric` **(primary)** — range: percent
  - Aggregated score across all tasks, typically computed as accuracy for classification tasks, F1 for paraphrase/QQP, and Pearson correlation for STS-B, following standard GLUE/SuperGLUE protocols.
- `similarity score` — range: [1, 5]
  - Continuous score from 1 to 5 predicting semantic similarity for sentence pairs in STS-B.

## Input / output format

**Input**: Sentence pairs, single sentences, or context-question-answer triples depending on the specific GLUE/SuperGLUE task variant.

**Output**: Class labels (e.g., acceptability, sentiment, entailment/contradiction/neutral, yes/no), similarity scores (1-5), or selected entities/answers.

## Scoring recipe

```python
def compute_glue_score(predictions, gold_labels, task_type):
    if task_type in ['classification', 'NLI', 'QA']:
        return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    elif task_type == 'paraphrase':
        return f1_score(gold_labels, predictions, average='binary')
    elif task_type == 'STS':
        return pearsonr(gold_labels, predictions)[0]
    return 0.0

# Aggregate across all PORTULAN ExtraGLUE tasks
final_score = mean([compute_glue_score(preds, golds, t) for t, preds, golds in tasks])
```

## Common pitfalls

- Translation-induced linguistic artifacts, such as gender bias and idiom misrepresentation, may skew performance on Portuguese variants compared to English baselines.
- The benchmark aggregates diverse task types (classification, regression, NLI, QA) into a single score, which can mask poor performance on specific linguistic phenomena.
- Low-rank adaptation (LoRA) fine-tuning on Albertina may underfit complex reasoning tasks compared to full fine-tuning, affecting cross-task comparability.

## Evidence (verbatim from paper)

> Both GLUE and SuperGLUE are aggregations of existing public datasets accompanied by a single-number performance metric and an analysis toolkit. The Semantic Textual Similarity Benchmark (STS-B)G (Cer et al.,, [2017])* is a task for predicting a similarity score (from 1 to 5) for each sentence pair.

## Citation

```bibtex
@misc{osorio2024portulanextraglue,
  title={PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese},
  author={Tomás Freitas Osório et al. (2024)},
  year={2024},
  note={arXiv:2404.05333}
}
```

- arXiv: 2404.05333