Blimp Glue Superglue Eval

---
name: blimp-glue-superglue-eval
description: Evaluates language understanding, linguistic acceptability, sentiment analysis, natural language inference, and factual reasoning under low-resource fine-tuning conditions. Use when the user wants to benchmark on BLiMP, GLUE (subset), SuperGLUE (subset), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.00544
  bibtex_key: [REDACTED]
  confidence: high
---

# blimp-glue-superglue-eval

> PaPaformer: Language Model from Pre-trained Parallel Paths — Tapaninaho et al. (2025) (arXiv:2508.00544, 2025)

## What this evaluates

Evaluates language understanding, linguistic acceptability, sentiment analysis, natural language inference, and factual reasoning under low-resource fine-tuning conditions.

## Datasets

- **BLiMP, GLUE (subset), SuperGLUE (subset)** — total ?; splits: train (-1), val (-1), test (-1); HF `blimp, glue, super_glue`; repo https://github.com/babylm/evaluation-pipeline-2024

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard task-level accuracy calculated as the proportion of correct predictions. Scores are averaged across multiple random seeds (typically 5) to mitigate initialization variance.

## Input / output format

**Input**: Text sequences or sentence pairs from BLiMP, GLUE, and SuperGLUE tasks, formatted according to the babylm/evaluation-pipeline-2024.

**Output**: Predicted class labels for each task.

## Scoring recipe

```python
import numpy as np
def compute_accuracy(predictions, gold_labels):
    return np.mean(np.array(predictions) == np.array(gold_labels))
seed_scores = []
for seed in seeds:
    preds, golds = run_finetune_and_evaluate(seed)
    seed_scores.append(compute_accuracy(preds, golds))
final_score = np.mean(seed_scores)
```

## Common pitfalls

- High sensitivity to random weight initialization seeds, especially for small models and small datasets, causing up to 12% metric variance.
- Routing/path selection in parallel architectures does not consistently align with optimal task performance; dominant path usage often correlates better than balanced routing.
- Qualitative generation evaluation was restricted to narrative prompts due to poor model performance on math/instructional prompts.

## Evidence (verbatim from paper)

> Model evaluations closely followed the methodologies provided in the babylm/evaluation-pipeline-2024 (Choshen et al., [2024]). However, not all tasks were included in the more lightweight evaluation pipeline, which this work followed. This pipeline applied the full BLiMP (Warstadt et al., [2020]) benchmark and a selected subset of tasks from the GLUE (Wang et al., [2018]) and SuperGLUE (Wang et al., [2019]) benchmarks... To address this problem, the final evaluation included multiple fine-tuning runs (5 different seeds: 3, 12, 42, 100, 200) for each model, and the reported results are averaged scores across runs. ... Table[6] exhibits an interesting perspective of the selection and utilization of parallel-model path... Accuracy | --- | 40% | 46.5% | 50

## Citation

```bibtex
@misc{tapaninaho2025papapformer,
  title={PaPaformer: Language Model from Pre-trained Parallel Paths},
  author={Tapaninaho et al. (2025)},
  year={2025},
  note={arXiv:2508.00544}
}
```

- arXiv: 2508.00544