ML inference latency optimization, model compression, distillation, caching strategies, and edge deployment patterns. Use when optimizing inference performance, reducing model size, or deploying ML at the edge.
Scanned 2/12/2026
Install via CLI
Skills Directoryopenskills install melodic-software/claude-code-plugins---
name: ml-inference-optimization
description: ML inference latency optimization, model compression, distillation, caching strategies, and edge deployment patterns. Use when optimizing inference performance, reducing model size, or deploying ML at the edge.
allowed-tools: Read, Glob, Grep
---
# ML Inference Optimization
## When to Use This Skill
Use this skill when:
- Optimizing ML inference latency
- Reducing model size for deployment
- Implementing model compression techniques
- Designing inference caching strategies
- Deploying models at the edge
- Balancing accuracy vs. latency trade-offs
**Keywords:** inference optimization, latency, model compression, distillation, pruning, quantization, caching, edge ML, TensorRT, ONNX, model serving, batching, hardware acceleration
## Inference Optimization Overview
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Inference Optimization Stack │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Model Level │ │
│ │ Distillation │ Pruning │ Quantization │ Architecture Search │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Compiler Level │ │
│ │ Graph optimization │ Operator fusion │ Memory planning │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Runtime Level │ │
│ │ Batching │ Caching │ Async execution │ Multi-threading │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Hardware Level │ │
│ │ GPU │ TPU │ NPU │ CPU SIMD │ Custom accelerators │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
## Model Compression Techniques
### Technique Overview
| Technique | Size Reduction | Speed Improvement | Accuracy Impact |
| --------- | -------------- | ----------------- | --------------- |
| **Quantization** | 2-4x | 2-4x | Low (1-2%) |
| **Pruning** | 2-10x | 1-3x | Low-Medium |
| **Distillation** | 3-10x | 3-10x | Medium |
| **Low-rank factorization** | 2-5x | 1.5-3x | Low-Medium |
| **Weight sharing** | 10-100x | Variable | Medium-High |
### Knowledge Distillation
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Knowledge Distillation │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ │
│ │ Teacher Model│ (Large, accurate, slow) │
│ │ GPT-4 │ │
│ └──────────────┘ │
│ │ │
│ ▼ Soft labels (probability distributions) │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Training Process │ │
│ │ Loss = α × CrossEntropy(student, hard_labels) │ │
│ │ + (1-α) × KL_Div(student, teacher_soft_labels) │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │Student Model │ (Small, nearly as accurate, fast) │
│ │ DistilBERT │ │
│ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
**Distillation Types:**
| Type | Description | Use Case |
| ---- | ----------- | -------- |
| **Response distillation** | Match teacher outputs | General compression |
| **Feature distillation** | Match intermediate layers | Better transfer |
| **Relation distillation** | Match sample relationships | Structured data |
| **Self-distillation** | Model teaches itself | Regularization |
### Pruning Strategies
```text
Unstructured Pruning (Weight-level):
Before: [0.1, 0.8, 0.2, 0.9, 0.05, 0.7]
After: [0.0, 0.8, 0.0, 0.9, 0.0, 0.7] (50% sparse)
• Flexible, high sparsity possible
• Needs sparse hardware/libraries
Structured Pruning (Channel/Layer-level):
Before: ┌───┬───┬───┬───┐
│ C1│ C2│ C3│ C4│
└───┴───┴───┴───┘
After: ┌───┬───┬───┐
│ C1│ C3│ C4│ (Removed C2 entirely)
└───┴───┴───┘
• Works with standard hardware
• Lower compression ratio
```
**Pruning Decision Criteria:**
| Method | Description | Effectiveness |
| ------ | ----------- | ------------- |
| **Magnitude-based** | Remove smallest weights | Simple, effective |
| **Gradient-based** | Remove low-gradient weights | Better accuracy |
| **Second-order** | Use Hessian information | Best but expensive |
| **Lottery ticket** | Find winning subnetwork | Theoretical insight |
### Quantization (Detailed)
```text
Precision Hierarchy:
FP32 (32 bits): ████████████████████████████████
FP16 (16 bits): ████████████████
BF16 (16 bits): ████████████████ (different mantissa/exponent)
INT8 (8 bits): ████████
INT4 (4 bits): ████
Binary (1 bit): █
Memory and Compute Scale Proportionally
```
**Quantization Approaches:**
| Approach | When Applied | Quality | Effort |
| -------- | ------------ | ------- | ------ |
| **Dynamic quantization** | Runtime | Good | Low |
| **Static quantization** | Post-training with calibration | Better | Medium |
| **QAT** | During training | Best | High |
## Compiler-Level Optimization
### Graph Optimization
```text
Original Graph:
Input → Conv → BatchNorm → ReLU → Conv → BatchNorm → ReLU → Output
Optimized Graph (Operator Fusion):
Input → FusedConvBNReLU → FusedConvBNReLU → Output
Benefits:
• Fewer kernel launches
• Better memory locality
• Reduced memory bandwidth
```
### Common Optimizations
| Optimization | Description | Speedup |
| ------------ | ----------- | ------- |
| **Operator fusion** | Combine sequential ops | 1.2-2x |
| **Constant folding** | Pre-compute constants | 1.1-1.5x |
| **Dead code elimination** | Remove unused ops | Variable |
| **Layout optimization** | Optimize tensor memory layout | 1.1-1.3x |
| **Memory planning** | Optimize buffer allocation | 1.1-1.2x |
### Optimization Frameworks
| Framework | Vendor | Best For |
| --------- | ------ | -------- |
| **TensorRT** | NVIDIA | NVIDIA GPUs, lowest latency |
| **ONNX Runtime** | Microsoft | Cross-platform, broad support |
| **OpenVINO** | Intel | Intel CPUs/GPUs |
| **Core ML** | Apple | Apple devices |
| **TFLite** | Google | Mobile, embedded |
| **Apache TVM** | Open source | Custom hardware, research |
## Runtime Optimization
### Batching Strategies
```text
No Batching:
Request 1: [Process] → Response 1 10ms
Request 2: [Process] → Response 2 10ms
Request 3: [Process] → Response 3 10ms
Total: 30ms, GPU underutilized
Dynamic Batching:
Requests 1-3: [Wait 5ms] → [Process batch] → Responses
Total: 15ms, 2x throughput
Trade-off: Latency vs. Throughput
• Larger batch: Higher throughput, higher latency
• Smaller batch: Lower latency, lower throughput
```
**Batching Parameters:**
| Parameter | Description | Trade-off |
| --------- | ----------- | --------- |
| `batch_size` | Maximum batch size | Throughput vs. latency |
| `max_wait_time` | Wait time for batch fill | Latency vs. efficiency |
| `min_batch_size` | Minimum before processing | Latency predictability |
### Caching Strategies
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Inference Caching Layers │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Layer 1: Input Cache │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Cache exact inputs → Return cached outputs │ │
│ │ Hit rate: Low (inputs rarely repeat exactly) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Layer 2: Embedding Cache │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Cache computed embeddings for repeated tokens/entities │ │
│ │ Hit rate: Medium (common tokens repeat) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Layer 3: KV Cache (for transformers) │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Cache key-value pairs for attention │ │
│ │ Hit rate: High (reuse across tokens in sequence) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Layer 4: Result Cache │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Cache semantic equivalents (fuzzy matching) │ │
│ │ Hit rate: Variable (depends on query distribution) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
**Semantic Caching for LLMs:**
```text
Query: "What's the capital of France?"
↓
Hash + Embed query
↓
Search cache (similarity > threshold)
↓
├── Hit: Return cached response
└── Miss: Generate → Cache → Return
```
### Async and Parallel Execution
```text
Sequential:
┌─────┐ ┌─────┐ ┌─────┐
│Prep │→│Model│→│Post │ Total: 30ms
│10ms │ │15ms │ │5ms │
└─────┘ └─────┘ └─────┘
Pipelined:
Request 1: │Prep│Model│Post│
Request 2: │Prep│Model│Post│
Request 3: │Prep│Model│Post│
Throughput: 3x higher
Latency per request: Same
```
## Hardware Acceleration
### Hardware Comparison
| Hardware | Strengths | Limitations | Best For |
| -------- | --------- | ----------- | -------- |
| **GPU (NVIDIA)** | High parallelism, mature ecosystem | Power, cost | Training, large batch inference |
| **TPU (Google)** | Matrix ops, cloud integration | Vendor lock-in | Google Cloud workloads |
| **NPU (Apple/Qualcomm)** | Power efficient, on-device | Limited models | Mobile, edge |
| **CPU** | Flexible, available | Slower for ML | Low-batch, CPU-bound |
| **FPGA** | Customizable, low latency | Development complexity | Specialized workloads |
### GPU Optimization
| Optimization | Description | Impact |
| ------------ | ----------- | ------ |
| **Tensor Cores** | Use FP16/INT8 tensor operations | 2-8x speedup |
| **CUDA graphs** | Reduce kernel launch overhead | 1.5-2x for small models |
| **Multi-stream** | Parallel execution | Higher throughput |
| **Memory pooling** | Reduce allocation overhead | Lower latency variance |
## Edge Deployment
### Edge Constraints
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Edge Deployment Constraints │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Resource Constraints: │
│ ├── Memory: 1-4 GB (vs. 64+ GB cloud) │
│ ├── Compute: 1-10 TOPS (vs. 100+ TFLOPS cloud) │
│ ├── Power: 5-15W (vs. 300W+ cloud) │
│ └── Storage: 16-128 GB (vs. TB cloud) │
│ │
│ Operational Constraints: │
│ ├── No network (offline operation) │
│ ├── Variable ambient conditions │
│ ├── Infrequent updates │
│ └── Long deployment lifetime │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
### Edge Optimization Strategies
| Strategy | Description | Use When |
| -------- | ----------- | -------- |
| **Model selection** | Use edge-native models (MobileNet, EfficientNet) | Accuracy acceptable |
| **Aggressive quantization** | INT8 or lower | Memory/power constrained |
| **On-device distillation** | Distill to tiny model | Extreme constraints |
| **Split inference** | Edge preprocessing, cloud inference | Network available |
| **Model caching** | Cache results locally | Repeated queries |
### Edge ML Frameworks
| Framework | Platform | Features |
| --------- | -------- | -------- |
| **TensorFlow Lite** | Android, iOS, embedded | Quantization, delegates |
| **Core ML** | iOS, macOS | Neural Engine optimization |
| **ONNX Runtime Mobile** | Cross-platform | Broad model support |
| **PyTorch Mobile** | Android, iOS | Familiar API |
| **TensorRT** | NVIDIA Jetson | Maximum performance |
## Latency Profiling
### Profiling Methodology
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Latency Breakdown Analysis │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Data Loading: ████████░░░░░░░░░░ 15% │
│ 2. Preprocessing: ██████░░░░░░░░░░░░ 10% │
│ 3. Model Inference: ████████████████░░ 60% │
│ 4. Postprocessing: ████░░░░░░░░░░░░░░ 8% │
│ 5. Response Serialization:███░░░░░░░░░░░░░░░ 7% │
│ │
│ Target: Model inference (60% = biggest optimization opportunity) │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
### Profiling Tools
| Tool | Use For |
| ---- | ------- |
| **PyTorch Profiler** | PyTorch model profiling |
| **TensorBoard** | TensorFlow visualization |
| **NVIDIA Nsight** | GPU profiling |
| **Chrome Tracing** | General timeline visualization |
| **perf** | CPU profiling |
### Key Metrics
| Metric | Description | Target |
| ------ | ----------- | ------ |
| **P50 latency** | Median latency | < SLA |
| **P99 latency** | Tail latency | < 2x P50 |
| **Throughput** | Requests/second | Meet demand |
| **GPU utilization** | Compute usage | > 80% |
| **Memory bandwidth** | Memory usage | < limit |
## Optimization Workflow
### Systematic Approach
```text
┌─────────────────────────────────────────────────────────────────────┐
│ Optimization Workflow │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Baseline │
│ └── Measure current performance (latency, throughput, accuracy) │
│ │
│ 2. Profile │
│ └── Identify bottlenecks (model, data, system) │
│ │
│ 3. Optimize (in order of effort/impact): │
│ ├── Hardware: Use right accelerator │
│ ├── Compiler: Enable optimizations (TensorRT, ONNX) │
│ ├── Runtime: Batching, caching, async │
│ ├── Model: Quantization, pruning │
│ └── Architecture: Distillation, model change │
│ │
│ 4. Validate │
│ └── Verify accuracy maintained, latency improved │
│ │
│ 5. Deploy and Monitor │
│ └── Track real-world performance │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
### Optimization Priority Matrix
```text
High Impact
│
Compiler Opts ────┼──── Quantization
(easy win) │ (best ROI)
│
Low Effort ──────────────┼──────────────── High Effort
│
Batching ────┼──── Distillation
(quick win) │ (major effort)
│
Low Impact
```
## Common Patterns
### Multi-Model Serving
```text
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ Request → ┌─────────┐ │
│ │ Router │ │
│ └─────────┘ │
│ │ │ │ │
│ ┌────────┘ │ └────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────┐ ┌───────┐ ┌───────┐ │
│ │ Tiny │ │ Small │ │ Large │ │
│ │ <10ms │ │ <50ms │ │<500ms │ │
│ └───────┘ └───────┘ └───────┘ │
│ │
│ Routing strategies: │
│ • Complexity-based: Simple→Tiny, Complex→Large │
│ • Confidence-based: Try Tiny, escalate if low confidence │
│ • SLA-based: Route based on latency requirements │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
### Speculative Execution
```text
Query: "Translate: Hello"
│
├──▶ Small model (draft): "Bonjour" (5ms)
│
└──▶ Large model (verify): Check "Bonjour" (10ms parallel)
│
├── Accept: Return immediately
└── Reject: Generate with large model
Speedup: 2-3x when drafts are often accepted
```
### Cascade Models
```text
Input → ┌────────┐
│ Filter │ ← Cheap filter (reject obvious negatives)
└────────┘
│ (candidates only)
▼
┌────────┐
│ Stage 1│ ← Fast model (coarse ranking)
└────────┘
│ (top-100)
▼
┌────────┐
│ Stage 2│ ← Accurate model (fine ranking)
└────────┘
│ (top-10)
▼
Output
Benefit: 10x cheaper, similar accuracy
```
## Optimization Checklist
### Pre-Deployment
- [ ] Profile baseline performance
- [ ] Identify primary bottleneck (model, data, system)
- [ ] Apply compiler optimizations (TensorRT, ONNX)
- [ ] Evaluate quantization (INT8 usually safe)
- [ ] Tune batch size for target throughput
- [ ] Test accuracy after optimization
### Deployment
- [ ] Configure appropriate hardware
- [ ] Enable caching where applicable
- [ ] Set up monitoring (latency, throughput, errors)
- [ ] Configure auto-scaling policies
- [ ] Implement graceful degradation
### Post-Deployment
- [ ] Monitor p99 latency
- [ ] Track accuracy metrics
- [ ] Analyze cache hit rates
- [ ] Review cost efficiency
- [ ] Plan iterative improvements
## Related Skills
- `llm-serving-patterns` - LLM-specific serving optimization
- `ml-system-design` - End-to-end ML pipeline design
- `quality-attributes-taxonomy` - Performance as quality attribute
- `estimation-techniques` - Capacity planning for ML systems
## Version History
- v1.0.0 (2025-12-26): Initial release - ML inference optimization patterns
---
## Last Updated
**Date:** 2025-12-26
No comments yet. Be the first to comment!
