Efficient Bert Eval

---
name: efficient-bert-eval
description: Evaluates the performance of efficiently trained BERT models (via Mixture-of-Supernets) on downstream natural language understanding tasks. It probes the trade-off between model size, training compute, and accuracy compared to standalone pretraining and other NAS baselines. Use when the user wants to benchmark on GLUE benchmark, or asks about evaluating this task. Reports Avg. GLUE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.04845
  bibtex_key: jawahar2023mixtureofsupernets
  confidence: high
---

# efficient-bert-eval

> Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts — Jawahar et al. (2023) (arXiv:2306.04845, 2023)

## What this evaluates

Evaluates the performance of efficiently trained BERT models (via Mixture-of-Supernets) on downstream natural language understanding tasks. It probes the trade-off between model size, training compute, and accuracy compared to standalone pretraining and other NAS baselines.

## Datasets

- **GLUE benchmark** — total ?; splits: validation (-1); HF `glue`

## Metrics

- `Avg. GLUE` **(primary)** — range: percent
  - Average accuracy across seven GLUE tasks: MNLI, CoLA, MRPC, SST2, QNLI, QQP, and RTE.

## Input / output format

**Input**: Text sequences for masked language modeling during pretraining; task-specific input pairs or single sequences for finetuning on GLUE tasks.

**Output**: Class predictions for each GLUE task, evaluated as accuracy.

## Scoring recipe

```python
accuracies = []
for task in ["MNLI", "CoLA", "MRPC", "SST2", "QNLI", "QQP", "RTE"]:
    preds = model.predict(task_data)
    acc = accuracy_score(task_labels, preds)
    accuracies.append(acc)
avg_glue = sum(accuracies) / len(accuracies)
```

## Common pitfalls

- Comparing against baselines that use additional pretraining steps (e.g., NAS-BERT uses 125K steps) without accounting for the compute difference.
- Confusing AutoDistil's proxy-search mode (which uses MNLI validation scores to rank architectures) with the agnostic-search mode used for fair comparison.

## Evidence (verbatim from paper)

> We evaluate the performance of the BERT model by finetuning on each of the seven tasks (chosen by AutoDistil) in the GLUE benchmark. Table 2 displays the GLUE benchmark performance of standalone training of the architecture (1x pretraining budget, which is 2048 batch size * 125,000 steps) as well as architecture-specific weights from different supernets (0 additional pretraining steps; that is, only supernet pretraining). On average GLUE, neuron-wise MoS can perform similarly or improves over NAS-BERT for different model sizes without any additional training.

## Citation

```bibtex
@misc{jawahar2023mixtureofsupernets,
  title={Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts},
  author={Jawahar et al. (2023)},
  year={2023},
  note={arXiv:2306.04845}
}
```

- arXiv: 2306.04845