Back to skills
SKILL.md
Sparseforge Hessian Mask
ASecurityEfficient semi-structured LLM sparsification via annealing of Hessian-mask guided pruning. Achieves high sparsity with minimal accuracy loss using second-order importance estimation.
- 3 stars
- 0 votes
- 0 copies
- 7 views
- Added September 11, 2026
Security analysis
100/100npx -y skills add hiyenwong/ai_collection --skill sparseforge-hessian-mask --agent claude-codeAre you the author of Sparseforge Hessian Mask?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/hiyenwong-sparseforge-hessian-mask)---
name: sparseforge-hessian-mask
description: Efficient semi-structured LLM sparsification via annealing of Hessian-mask guided pruning. Achieves high sparsity with minimal accuracy loss using second-order importance estimation.
category: deep-learning
tags: [LLM, sparsification, pruning, Hessian, compression, efficiency]
trigger: sparseforge, hessian mask, semi-structured pruning, LLM sparsification, 2:4 sparsity, model compression
---
# SparseForge: Hessian-Mask Guided Semi-Structured Pruning
## Overview
SparseForge achieves efficient semi-structured (e.g., 2:4) sparsity in LLMs by using Hessian-based second-order importance estimation with an annealing schedule to guide pruning decisions.
## Core Technique
1. **Hessian-Driven Importance**: Compute diagonal Hessian (or Fisher information) to estimate parameter sensitivity
2. **Mask Annealing**: Instead of one-shot pruning, gradually anneal a binary mask over training steps
3. **Semi-Structured Constraints**: Enforce N:M sparsity patterns (e.g., 2 out of every 4 weights = 0) for hardware efficiency
4. **Recovery Fine-Tuning**: Brief fine-tuning with the annealed mask to recover accuracy
## Key Benefits
- **Higher accuracy** at same sparsity vs. magnitude-based pruning
- **Hardware-friendly** 2:4 sparsity patterns compatible with Tensor Cores
- **Annealing** prevents catastrophic accuracy drops from aggressive one-shot pruning
## Implementation Steps
1. Compute per-parameter Hessian diagonal via Fisher approximation (gradient outer product)
2. Initialize soft mask with sigmoid parametrization
3. Anneal mask temperature over training: soft → hard binary decisions
4. Apply N:M structural constraints during mask hardening
5. Fine-tune with masked weights, optionally with distillation loss from dense model
## Pitfalls
- Hessian computation is expensive — use diagonal/Fisher approximation for LLM-scale models
- Annealing schedule is critical: too fast → accuracy collapse; too slow → wasted compute
- N:M constraints require specialized kernel support or careful indexing for inference speedup
## Activation Keywords
sparseforge, hessian mask, semi-structured pruning, LLM sparsification, 2:4 sparsity, model compression, Fisher information, mask annealing
Attribution
Related skills
View all in tools →Comments
Loading comments…
Newsletter
Get the weekly list of safe new skills, and the ones we pulled.