Deep Compression Eval

---
name: deep-compression-eval
description: Evaluates the effectiveness of a three-stage neural network compression pipeline (pruning, trained quantization, and Huffman coding) in reducing model storage size while preserving classification accuracy on standard computer vision benchmarks. Use when the user wants to benchmark on MNIST, ImageNet (ILSVRC-2012), or asks about evaluating this task. Reports Top-1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1510.00149
  bibtex_key: han2015deepcompression
  confidence: high
---

# deep-compression-eval

> Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding — Song Han et al. (2015) (arXiv:1510.00149, 2015)

## What this evaluates

Evaluates the effectiveness of a three-stage neural network compression pipeline (pruning, trained quantization, and Huffman coding) in reducing model storage size while preserving classification accuracy on standard computer vision benchmarks.

## Datasets

- **MNIST** — total ?; splits: (unstated)
- **ImageNet (ILSVRC-2012)** — total 1250000; splits: train (1200000), val (50000)

## Metrics

- `Top-1 Accuracy` **(primary)** — range: percent
  - Percentage of images where the predicted class matches the ground truth class. Calculated as (correct predictions / total images) * 100.
- `Compress Rate` — range: other
  - Ratio of the original model parameter storage size to the final compressed file size, including overhead from the weight codebook and sparse index arrays.

## Input / output format

**Input**: Raw image pixels fed into a standard Caffe neural network architecture (LeNet-300-100, LeNet-5, AlexNet, or VGG-16).

**Output**: Predicted class label for each input image.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, original_model, compressed_model_file):
    # Top-1 Accuracy
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    top1_acc = (correct / len(gold_labels)) * 100
    
    # Compress Rate
    original_size_bytes = sum(p.numel() for p in original_model.parameters()) * 4  # float32 baseline
    compressed_size_bytes = len(compressed_model_file)
    compress_rate = original_size_bytes / compressed_size_bytes
    
    return top1_acc, compress_rate
```

## Common pitfalls

- Reporting compression as a percentage reduction instead of the multiplier (e.g., 35×) used in the paper.
- Failing to account for the storage overhead of the codebook and sparse index arrays when calculating the final compressed size.
- Assuming accuracy drops are acceptable without performing the specified fine-tuning step after pruning and quantization.

## Evidence (verbatim from paper)

> We use the AlexNet Caffe model as the reference model, which has 61 million parameters and achieved a top-1 accuracy of 57.2% and a top-5 accuracy of 80.3%. The network parameters and accuracy before and after pruning are shown in Table 1. The compression pipeline saves network storage by 35× to 49× across different networks without loss of accuracy.

## Citation

```bibtex
@misc{han2015deepcompression,
  title={Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding},
  author={Song Han et al. (2015)},
  year={2015},
  note={arXiv:1510.00149}
}
```

- arXiv: 1510.00149