S3

---
name: S3
description: Work with S3-compatible object storage with proper security, lifecycle policies, and access patterns.
metadata: {"clawdbot":{"emoji":"🪣","os":["linux","darwin","win32"]}}
---

## Public Access Control

- Default deny public access—only open when explicitly needed (static hosting)
- Bucket policy vs IAM: bucket policy for cross-account/public, IAM for same-account roles
- Check both bucket-level AND account-level block settings—account can override bucket
- For web assets, prefer CDN in front of bucket over direct public access

## Presigned URLs

- Set shortest expiration practical—minutes for immediate use, not days
- URL is a bearer token—anyone with it has access; treat as secret
- Specify HTTP method in signature—GET for download, PUT for upload
- Include Content-Type for uploads—mismatch between signature and request causes 403
- Generate server-side, never expose credentials to client

## Lifecycle Rules

- Transition to cheaper tiers for infrequent access—but check minimum storage duration penalties
- Auto-delete for temp files, logs, old versions—prevents unbounded storage growth
- Clean incomplete multipart uploads—accumulate invisibly; set abort rule (7 days typical)
- Versioned buckets: separate rules for current vs noncurrent versions

## Versioning Behavior

- Enable before you need it—can't recover deleted objects without versioning
- "Delete" creates delete marker—object hidden but versions remain; storage still consumed
- Permanent deletion requires explicit version ID—without it, just adds marker
- Noncurrent version expiration essential—otherwise old versions accumulate forever

## Multipart Uploads

- Required above 5GB, recommended above 100MB—single PUT has size limits
- Incomplete uploads invisible in normal listings—consume storage silently
- Abort incomplete uploads via lifecycle—or manually with `list-multipart-uploads`
- Parallel part uploads for speed—parts can upload concurrently

## CORS for Browser Access

- Required for JavaScript direct upload/download—blocked without CORS headers
- Specify exact origins—avoid wildcard `*` for authenticated requests
- Expose headers that JavaScript needs to read—Content-Length, ETag, custom headers
- AllowedMethods: GET for download, PUT for upload, DELETE if needed

## Key Naming

- Use prefixes like directories: `users/123/avatar.jpg`—but S3 is flat, not hierarchical
- Avoid sequential prefixes for high throughput—`2024-01-01/file1` can hotspot
- Random prefix or hash for write-heavy buckets—distributes across partitions
- No leading slash—`/images/file.jpg` creates empty-string prefix

## Cost Awareness

- Request volume matters—many small files more expensive than few large files
- Egress typically costly—CDN reduces egress by caching at edge
- Minimum storage duration varies by tier—early deletion still charged full period
- Lifecycle transitions have per-object cost—millions of tiny files expensive to transition

## Replication

- Cross-region for disaster recovery, same-region for compliance copies
- Versioning required on both source and destination
- Only new objects replicate—existing objects need manual copy or batch operation
- Delete markers not replicated by default—explicitly enable if needed

## Provider Differences

- AWS S3: full feature set, most tools assume AWS behavior
- Cloudflare R2: no egress fees, subset of features
- Backblaze B2: S3-compatible API, different pricing model
- MinIO: self-hosted, full S3 API compatibility
- Check presigned URL compatibility—some providers have quirks