Consensus Layer Pruning Eval

---
name: consensus-layer-pruning-eval
description: Evaluates a multi-metric layer pruning method (Consensus) on image classification models, measuring trade-offs between computational efficiency (FLOPs reduction) and predictive performance (accuracy drop), while also assessing robustness against adversarial and out-of-distribution attacks. Use when the user wants to benchmark on CIFAR-10, ImageNet, CIFAR-10.2, CIFAR-C, ImageNet-C, or asks about evaluating this task. Reports Δ Acc. (difference in accuracy).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.14345
  bibtex_key: [REDACTED]
  confidence: high
---

# consensus-layer-pruning-eval

> Layer Pruning with Consensus: A Triple-Win Solution — Giusti Mugnaini et al. (2024) (arXiv:2411.14345, 2024)

## What this evaluates

Evaluates a multi-metric layer pruning method (Consensus) on image classification models, measuring trade-offs between computational efficiency (FLOPs reduction) and predictive performance (accuracy drop), while also assessing robustness against adversarial and out-of-distribution attacks.

## Datasets

- **CIFAR-10** — total ?; splits: test (-1)
- **ImageNet** — total ?; splits: val (-1)
- **CIFAR-10.2** — total ?; splits: test (-1)
- **CIFAR-C** — total ?; splits: test (-1)
- **ImageNet-C** — total ?; splits: test (-1)

## Metrics

- `Δ Acc. (difference in accuracy)` **(primary)** — range: percent
  - Percentage point difference between pruned model accuracy and unpruned baseline accuracy: (Acc_pruned - Acc_original) * 100. Positive values indicate improvement, negative indicate drop.
- `FLOPs reduction` — range: percent
  - Percentage reduction in floating-point operations relative to the original unpruned model: ((FLOPs_original - FLOPs_pruned) / FLOPs_original) * 100.
- `Mean accuracy across benchmarks` — range: percent
  - Average classification accuracy computed across all specified adversarial/OOD benchmarks (CIFAR-10.2, CIFAR-C, ImageNet-C, FGSM) for a given pruning criterion.

## Input / output format

**Input**: Pruned ResNet architectures (ResNet32, ResNet44, ResNet50, ResNet56) evaluated on clean and adversarial/out-of-distribution image datasets.

**Output**: Tables reporting Δ Acc. (in percentage points) and FLOPs reduction (%) at matched compression levels, plus mean accuracy scores across robustness benchmarks.

## Scoring recipe

```python
def compute_delta_acc(original_acc, pruned_acc):
    return (pruned_acc - original_acc) * 100

def compute_flops_reduction(orig_flops, pruned_flops):
    return ((orig_flops - pruned_flops) / orig_flops) * 100

def compute_robustness_acc(model, benchmarks):
    accs = [model.evaluate(b) for b in benchmarks]
    return sum(accs) / len(accs)
```

## Common pitfalls

- Reporting absolute accuracy instead of the required delta accuracy (Δ Acc.) relative to the unpruned baseline.
- Comparing pruning methods at mismatched FLOP reduction percentages without exact matching or interpolation.
- Using incorrect severity levels for CIFAR-C (severity 4) and ImageNet-C (severity 5) when computing robustness metrics.

## Evidence (verbatim from paper)

> To assess the predictive ability of the unpruned models against their pruned versions, we adhere to standard practices by reporting the difference in accuracy [20, 42]. For the semantic-preserving attacks CIFAR-C and ImageNet-C, we report the average across all possible attacks as suggested by previous work [21]. Regardless of the dataset, negative values mean a decrease in accuracy, while positive values denote an improvement, both measured in percentage points (pp).

## Citation

```bibtex
@misc{giustimugnaini2024layerpruning,
  title={Layer Pruning with Consensus: A Triple-Win Solution},
  author={Giusti Mugnaini et al. (2024)},
  year={2024},
  note={arXiv:2411.14345}
}
```

- arXiv: 2411.14345