Hies Pruning Eval

---
name: hies-pruning-eval
description: Evaluates the accuracy and stability of transformer head pruning methods (specifically HIES vs. baselines) across NLP, vision, and multimodal benchmarks at fixed sparsity ratios (10%, 30%, 50%). Use when the user wants to benchmark on GLUE (SST-2, CoLA, MRPC, QQP, STS-B, QNLI, MNLI, RTE), HellaSwag, Winogrande, ARC-e / ARC-c, OBQA, ImageNet1k, CIFAR-100, Food-101, Fashion MNIST, VizWiz-VQA, MM-Vet, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13832
  bibtex_key: choi2025hies
  confidence: high
---

# hies-pruning-eval

> Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning — Choi et al. (2025) (arXiv:2510.13832, 2025)

## What this evaluates

Evaluates the accuracy and stability of transformer head pruning methods (specifically HIES vs. baselines) across NLP, vision, and multimodal benchmarks at fixed sparsity ratios (10%, 30%, 50%).

## Datasets

- **GLUE (SST-2, CoLA, MRPC, QQP, STS-B, QNLI, MNLI, RTE)** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **Winogrande** — total ?; splits: test (-1)
- **ARC-e / ARC-c** — total ?; splits: test (-1)
- **OBQA** — total ?; splits: test (-1)
- **ImageNet1k** — total ?; splits: test (-1)
- **CIFAR-100** — total ?; splits: test (-1)
- **Food-101** — total ?; splits: test (-1)
- **Fashion MNIST** — total ?; splits: test (-1)
- **VizWiz-VQA** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1] | percent
  - Task-specific metric: Accuracy for most tasks, Matthews correlation coefficient for CoLA, F1 score for MRPC, and Pearson correlation for STS-B. Averaged across tasks for the 'Average' column.
- `Stability` — range: percent
  - Percentage of the pruned model's task metric relative to the unpruned baseline's metric, indicating robustness to aggressive compression.

## Input / output format

**Input**: Pruned transformer model (BERT_base, LLaMA-2_7B, ViT_Large, LLaVA-1.5_7B) fed with benchmark inputs (text, images, or image-text pairs).

**Output**: Task predictions (class labels, scores, or generated text) used to compute benchmark metrics.

## Scoring recipe

```python
def compute_accuracy(preds, gold, task):
    if task == 'CoLA': return matthews_corrcoef(gold, preds)
    if task == 'MRPC': return f1_score(gold, preds)
    if task == 'STS-B': return pearsonr(gold, preds)
    return accuracy_score(gold, preds)

def compute_stability(pruned_acc, base_acc):
    return (pruned_acc / base_acc) * 100
```

## Common pitfalls

- Stability is a relative metric (pruned vs. unpruned accuracy), not absolute performance.
- GLUE tasks use different evaluation metrics (Matthews corr, F1, Pearson corr, Accuracy) that are averaged in the 'Average' column.
- Pruning ratios are fixed at 10/30/50% for BERT but extended to 60% for LLaMA-2.

## Evidence (verbatim from paper)

> Datasets. We evaluate on various widely-adopted benchmarks: GLUE*[Wang et al., [2018]]*, HellaSwag*[Zellers et al., [2019]]*, Winogrande*[Sakaguchi et al., [2020]]*, the AI2 Reasoning Challenge—ARC-e/ARC-c*[Clark et al., [2018]]*, OBQA*[Mihaylov et al., [2018]]*, ImageNet1k*[Deng et al., [2009]]*, CIFAR-100*[Krizhevsky, [2009]]*, Food-101*[Bossard et al., [2014]]*, Fashion MNIST*[Xiao et al., [2017]]*, VizWiz-VQA*[Gurari et al., [2018]]*, and MM-Vet*[Yu et al., [2024]]*. We evaluate HIES using two key metrics: model quality and stability.

## Citation

```bibtex
@misc{choi2025hies,
  title={Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning},
  author={Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.13832}
}
```

- arXiv: 2510.13832