Design Scalable System Architecture

---
name: design-scalable-system-architecture
description: "Design scalable systems with sound trade-offs, clear boundaries, and maintainable patterns."
category: "Community"
author: community
version: "1.0.0"
icon: puzzle
---

# Software Architecture Rules

## Design Principles
- Simple until proven insufficient — complexity is a cost, not a feature
- Separate what changes from what stays stable — boundaries at change boundaries
- Design for the next 10x, not 100x — over-engineering wastes resources
- Make decisions reversible when possible — defer irreversible ones until necessary
- Constraints clarify design — embrace limitations, don't fight them early

## System Boundaries
- Define clear interfaces between components — contracts enable independent evolution
- Boundaries where teams split — Conway's Law is real, design with it
- Data ownership at boundaries — one source of truth per entity
- Async communication for loose coupling — sync calls create distributed monoliths
- Fail independently — one component's failure shouldn't cascade

## Trade-off Analysis
- Every decision has costs — articulate what you're giving up
- Consistency vs availability vs partition tolerance — pick two (CAP theorem)
- Performance vs maintainability — optimize hot paths, keep the rest readable
- Build vs buy — build differentiators, buy commodities
- Document the "why not" for rejected alternatives — future you needs context

## Scalability
- Stateless services scale horizontally — state makes scaling hard
- Cache aggressively, invalidate carefully — caching solves and creates problems
- Database is usually the bottleneck — read replicas, sharding, or denormalization
- Queue work that can be async — users don't need to wait for everything
- Scale for expected load, prepare for 3x spikes — headroom prevents outages

## Data Architecture
- Schema design constrains everything — get it right early, migrations are expensive
- Normalize for writes, denormalize for reads — optimize for access patterns
- Event sourcing when audit trail matters — reconstruct state from events
- CQRS when read/write patterns differ significantly — separate models for each
- Data gravity is real — processing moves to data, not vice versa

## Reliability
- Design for failure — everything fails eventually, handle it gracefully
- Timeouts on all external calls — hung connections cascade into outages
- Circuit breakers prevent cascade failures — fail fast, recover gradually
- Idempotency for retries — duplicate messages shouldn't corrupt state
- Graceful degradation over total failure — partial functionality beats error pages

## Security
- Defense in depth — multiple layers, no single point of failure
- Least privilege — minimal permissions for each component
- Encrypt in transit and at rest — assume networks and disks are hostile
- Validate at boundaries — don't trust input from outside your system
- Secrets management from day one — retrofitting is painful

## Evolution
- Design for replacement, not immortality — components will be rewritten
- Incremental migration over big bang — strangler fig pattern works
- Backwards compatibility for APIs — breaking changes break trust
- Feature flags decouple deploy from release — ship dark, enable gradually
- Monitor before, during, and after changes — data beats intuition

## Documentation
- Document decisions, not just structures — ADRs capture reasoning
- Diagrams at multiple zoom levels — C4 model: context, containers, components
- Keep docs near code — separate wikis go stale
- Update docs when architecture changes — wrong docs are worse than none
- Document operational aspects — runbooks, SLOs, failure modes

## Communication
- Translate technical decisions to business impact — stakeholders need context
- Present options with trade-offs — don't just recommend, explain
- Listen to operators — they know what breaks
- Involve security early — bolt-on security is weak security
- Decisions need buy-in — imposed architecture breeds resentment