Plm Structural Pruning Eval

---
name: plm-structural-pruning-eval
description: Evaluates structural pruning methods for pre-trained language models (BERT-base, RoBERTa-base) across eight text classification tasks. It measures the trade-off between model size (parameter count) and task performance (validation error) to identify Pareto-optimal sub-networks. Use when the user wants to benchmark on eight text classification tasks, or asks about evaluating this task. Reports Hypervolume.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.02267
  bibtex_key: klein2024structuralpruning
  confidence: high
---

# plm-structural-pruning-eval

> Structural Pruning of Pre-trained Language Models via Neural Architecture Search — Klein et al. (2024) (arXiv:2405.02267, 2024)

## What this evaluates

Evaluates structural pruning methods for pre-trained language models (BERT-base, RoBERTa-base) across eight text classification tasks. It measures the trade-off between model size (parameter count) and task performance (validation error) to identify Pareto-optimal sub-networks.

## Datasets

- **eight text classification tasks** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Hypervolume` **(primary)** — range: [0, 4]
  - HV(P_f|r) = λ(∪_{y∈P_f}[y, r]), where λ is the M-th dimensional Lebesgue measure. Objectives are first normalized via Quantile normalization to [0,1].

## Input / output format

**Input**: Text classification instances (input text and label) from eight tasks.

**Output**: Pruned sub-network architectures (binary masks for attention heads, feed-forward units, or layers) with fine-tuned weights, evaluated on validation/test sets.

## Scoring recipe

```python
def compute_hypervolume(pareto_set, reference_point=(2,2)):
    # 1. Quantile-normalize parameter count and error across all methods/repetitions to [0,1]
    # 2. Define reference point r = (2, 2)
    # 3. Compute Lebesgue measure of union of hyper-rectangles [y, r] for y in pareto_set
    # 4. Return HV value
    pass
```

## Common pitfalls

- Hypervolume must be computed on objectives normalized via Quantile normalization, not raw values.
- The reference point is fixed at (2, 2) after normalization, yielding a maximum possible HV of 4, not the standard (1, 1).
- Results are aggregated by averaging ranks across datasets and time steps, not just by final HV values.

## Evidence (verbatim from paper)

> To quantify the performance of a Pareto set $P_{f}$ , we compute for each Pareto set the Hypervolume (Zitzler et al., 2003) and report the regret, i.e. the difference to the best possible Hypervolume averaged across all repetitions. Given a reference point $\mathbf{r} \in \mathbb{R}^{M}$ , the Hypervolume $HV(P_{f}|\mathbf{r}) = \lambda (\cup_{\mathbf{y} \in P_{f}}[\mathbf{y},\mathbf{r}])$ is defined as the $M$ -th dimensional Lebesgue measure $\lambda$ between the Pareto set $P_{f}$ and the reference point $\mathbf{r}$ (see Figure 4 for an example). To compute the Hypervolume, we first normalize each objective based on all observed values across all methods and repetitions via Quantile normalization. This results in a uniform distribution between $[0,1]$ , and we use $r = (2,2)$ as reference point, which means the highest possible Hypervolume would be 4.

## Citation

```bibtex
@misc{klein2024structuralpruning,
  title={Structural Pruning of Pre-trained Language Models via Neural Architecture Search},
  author={Klein et al. (2024)},
  year={2024},
  note={arXiv:2405.02267}
}
```

- arXiv: 2405.02267