Glue Robustness Eval

---
name: glue-robustness-eval
description: This protocol evaluates whether language models memorize benchmark surface features or demonstrate true semantic robustness. It measures performance degradation when inputs are paraphrased, lexically/syntactically perturbed, or adversarially rewritten, contrasting deterministic greedy decoding with stochastic distributional evaluation. Use when the user wants to benchmark on GLUE (MNLI, QQP, QNLI, SST-2), or asks about evaluating this task. Reports GLUE robustness ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.07239
  bibtex_key: joshi2026stochasticchaos
  confidence: high
---

# glue-robustness-eval

> Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition — Joshi et al. (2026) (arXiv:2601.07239, 2026)

## What this evaluates

This protocol evaluates whether language models memorize benchmark surface features or demonstrate true semantic robustness. It measures performance degradation when inputs are paraphrased, lexically/syntactically perturbed, or adversarially rewritten, contrasting deterministic greedy decoding with stochastic distributional evaluation.

## Datasets

- **GLUE (MNLI, QQP, QNLI, SST-2)** — total ?; splits: orig (-1), para (-1), pert (-1), adv (-1); HF `glue`

## Metrics

- `GLUE robustness ratio` **(primary)** — range: [0, 1]
  - R_t^(d)(m) = (A_{t,para}^{(d)}(m) + A_{t,pert}^{(d)}(m) + A_{t,adv}^{(d)}(m)) / (3 * A_{t,orig}^{(d)}(m)), where A is the accuracy under decoding mode d on the respective dataset variant. Values range from 0 to 1 when variant performance does not exceed original performance.

## Input / output format

**Input**: Natural-language task description with constrained answer format (e.g., options A/B/C), applied to original, paraphrased, perturbed, or adversarially rewritten GLUE examples.

**Output**: Single text completion per prompt. Deterministic mode yields one completion; stochastic mode yields K=10 independent completions. A label-extraction function phi maps each completion to a task label.

## Scoring recipe

```python
for each task t, model m, decoding mode d in [Det, Stoch]:
  for each variant v in [orig, para, pert, adv]:
    generate completions for D_t^v
    extract labels phi(y_hat) for each completion
    if d == Stoch: majority_label = mode(phi(y_hat_1)...phi(y_hat_K))
    else: majority_label = phi(y_hat_det)
    A_t_v = mean(majority_label == gold_label)
R = (A_orig_para + A_orig_pert + A_orig_adv) / (3 * A_orig_orig)
return R
```

## Common pitfalls

- Reporting only deterministic accuracy hides distributional generalization; stochastic majority-vote often yields higher robustness scores.
- The robustness ratio assumes variant performance does not exceed original performance; if it does, R can exceed 1, complicating interpretation.
- Label extraction phi must be consistent across all variants; inconsistent parsing invalidates accuracy comparisons.

## Evidence (verbatim from paper)

> To isolate robustness rather than absolute accuracy, we define a GLUE robustness ratio for each triplet $(t,m,d)$: R_t^(d)(m) = (A_{t,para}^{(d)}(m) + A_{t,pert}^{(d)}(m) + A_{t,adv}^{(d)}(m)) / (3 A_{t,orig}^{(d)}(m)). By construction, R_t^(d)(m) \in [0,1] whenever the model performs no better on the variants than on the original split.

## Citation

```bibtex
@misc{joshi2026stochasticchaos,
  title={Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition},
  author={Joshi et al. (2026)},
  year={2026},
  note={arXiv:2601.07239}
}
```

- arXiv: 2601.07239