Storage

---
name: Storage
slug: storage
version: 1.0.0
description: Choose and architect storage systems for applications with the right tradeoffs.
homepage: https://clawic.com/skills/storage
metadata:
  clawdbot:
    emoji: 💾
    os:
    - linux
    - darwin
    - win32
    displayName: Storage
---

## Object vs Block vs File

- Object storage (S3, R2, GCS) for immutable blobs: images, videos, backups, logs — cheap, scales infinitely, but no partial updates
- Block storage (EBS, Persistent Disks) for databases and apps needing filesystem semantics — faster, but tied to single instance
- Network file systems (NFS, EFS) when multiple instances need shared filesystem access — convenient but latency and cost add up
- Default to object storage for user uploads — block storage for database files only

## When SQL vs NoSQL

- SQL when you need joins, transactions, or complex queries — fighting against NoSQL for relational data wastes months
- Document stores (MongoDB, Firestore) for nested/variable schemas where you always fetch the whole document
- Key-value (Redis, DynamoDB) for simple lookups by ID at massive scale — not for complex queries
- Time-series databases (InfluxDB, TimescaleDB) for metrics with timestamp-based queries — regular SQL struggles with retention policies
- Start with PostgreSQL unless you have a specific reason not to — it handles JSON, full-text search, and scales further than most assume

## Local vs Cloud Storage

- Local disk for ephemeral data: temp files, build artifacts, caches — assume it disappears on restart
- Cloud storage for anything that must survive instance termination — never store user data only on local disk
- Local SSD for databases in production — network-attached storage adds latency to every query
- Hybrid: local cache in front of cloud storage for frequently accessed files

## CDN Patterns

- Put CDN in front of static assets always — origin requests are slower and more expensive
- Set long cache TTLs with versioned URLs (`style.abc123.css`) — cache invalidation is slow and unreliable
- CDN for dynamic content only if latency matters more than freshness — adds complexity for marginal gains
- Edge caching for API responses works but cache keys get tricky — start simple, add only when needed

## Upload Handling

- Never accept uploads directly to app server disk in production — use presigned URLs to cloud storage
- Set file size limits at load balancer level, not just application — prevents memory exhaustion attacks
- Generate unique keys for uploads (UUIDs) — user-provided filenames cause collisions and path traversal risks
- Validate file types by content (magic bytes), not extension — extensions are trivially spoofed

## Data Locality

- Keep compute and storage in same region — cross-region data transfer adds latency and cost
- Replicate data to regions where users are, not where developers are
- Multi-region storage adds complexity — single region with backups elsewhere usually sufficient
- Database read replicas in user regions for read-heavy workloads

## Retention and Lifecycle

- Define retention policy before storing data — "keep everything" becomes expensive and legally risky
- Automate deletion of temporary data — manual cleanup never happens consistently
- Tiered storage for aging data: hot → warm → cold → archive — but check retrieval costs before archiving
- Separate storage for logs vs business data — different retention, different compliance requirements

## Cost Traps

- Egress fees dominate cloud storage costs — calculate before choosing provider
- Many small files cost more than few large files — batch small writes when possible
- Minimum storage duration on cold tiers — early deletion still charges full period
- API request costs matter at scale — millions of LIST operations add up

## Backup Strategy

- 3-2-1 rule: 3 copies, 2 different media types, 1 offsite — cloud counts as one location
- Test restores regularly — untested backups are not backups
- Point-in-time recovery for databases — daily snapshots lose a day of data
- Version important files — deletion or corruption often discovered late