Memcached

---
name: memcached
description: "Memcached technology expert. Deep expertise in distributed caching, slab allocation, consistent hashing, cache strategies, and operational tuning. WHEN: \"Memcached\", \"memcached\", \"memcache\", \"slab allocator\", \"cache invalidation\", \"consistent hashing\", \"memcached stats\", \"cache hit ratio\", \"LRU memcached\", \"mcrouter\", \"twemproxy\", \"ElastiCache Memcached\"."
license: MIT
---

# Memcached

This skill covers Memcached (current stable: 1.6.x). It covers Memcached internals -- slab allocation, LRU eviction, consistent hashing, multi-threaded architecture, protocol details, proxy layers (mcrouter, twemproxy), managed services (AWS ElastiCache for Memcached), and operational tuning. Memcached is a single-version-line technology; there are no version subdirectories.

## When to Use This Skill

**Use this agent for:**
- "How does Memcached slab allocation work?"
- "Tune Memcached memory for a session store workload"
- "Set up consistent hashing across a Memcached cluster"
- "Compare Memcached vs Redis for caching"
- "Diagnose high eviction rate in Memcached"
- "Configure mcrouter for connection pooling"
- "ElastiCache Memcached node sizing and auto-discovery"

**Route elsewhere when:**
- Question is about Redis-specific features (pub/sub, streams, sorted sets) --> the `redis` skill
- Question is about general caching paradigms --> the `overview` skill's `references/paradigm-keyvalue.md`
- Question compares multiple database technologies --> the `overview` skill

## How to Approach Tasks

When you receive a request:

1. **Classify** the request:
   - **Architecture/internals** -- Load `references/architecture.md`
   - **Performance diagnostics** -- Load `references/diagnostics.md`
   - **Configuration/operations** -- Load `references/best-practices.md`
   - **Comparison with other databases** -- see the `overview` skill

2. **Analyze** -- Apply Memcached-specific reasoning. Reference the slab allocator, multi-threaded model, LRU segmentation, and the fact that Memcached is volatile (no persistence). Every recommendation must account for the cache-only nature of the data.

3. **Recommend** -- Provide actionable guidance with specific startup flags, `stats` commands, client library configuration, or proxy-layer settings.

4. **Verify** -- Suggest validation steps (`stats`, `stats slabs`, `stats items`, hit ratio calculations, telnet probes).

## Core Expertise

### Slab Allocator

Memcached pre-allocates memory into **slab classes** to eliminate malloc fragmentation. This is the most important architectural concept:

- Memory is divided into **pages** (default 1 MB each)
- Each page belongs to a **slab class** with a fixed **chunk size**
- Chunk sizes grow by a **growth factor** (`-f`, default 1.25)
- Items are stored in the smallest chunk that fits (key + value + metadata ~56 bytes overhead)
- Once a page is assigned to a slab class, it cannot be reassigned (unless using `slab_reassign`)

**Default slab class progression (factor 1.25):**
```
Class  1: chunk size     96 bytes (base size)
Class  2: chunk size    120 bytes
Class  3: chunk size    152 bytes
Class  4: chunk size    192 bytes
Class  5: chunk size    240 bytes
...
Class 42: chunk size 1048576 bytes (1 MB, the max item size)
```

**Key parameters:**
```bash
-m 4096          # Maximum memory in MB (default 64)
-f 1.25          # Growth factor (smaller = more classes, less waste per item)
-n 48            # Minimum space for key+value+flags (chunk overhead added on top)
-I 1m            # Maximum item size (default 1 MB; can increase up to 128 MB)
-C               # Disable CAS (compare-and-swap) to save 8 bytes per item
```

**Slab calcification problem:**
- Over time, traffic patterns change but slab pages are permanently assigned
- Class A may have 80% of pages but only 20% of traffic
- Class B may have 5% of pages but 50% of traffic, causing evictions
- Solution: enable `slab_reassign` and `slab_automove` (default in 1.6.x)

```bash
# Enable automatic slab rebalancing
-o slab_reassign,slab_automove=1
# slab_automove=0: disabled
# slab_automove=1: slow, conservative rebalancing (recommended)
# slab_automove=2: aggressive rebalancing (use with caution)
```

### LRU Eviction (Segmented LRU)

Memcached 1.4.24+ uses a **segmented LRU** with separate queues per slab class:

| Queue | Purpose | Behavior |
|---|---|---|
| **HOT** | Recently written items | New items land here. Items age out to WARM on access or to COLD on timeout. |
| **WARM** | Actively accessed items | Items promoted from HOT or re-promoted from COLD. Eviction candidates age to COLD. |
| **COLD** | Eviction candidates | Least recently used items. Evictions happen from the tail of COLD. |
| **TEMP** | Short-TTL items (<61s by default) | Items with very short TTLs that should not pollute other queues. Never promoted. |

**LRU crawler and maintainer:**
```bash
# Enable the background LRU maintainer thread (default in 1.6.x)
-o lru_maintainer
# Enable the LRU crawler to reclaim expired items proactively
-o lru_crawler
# Tune TEMP queue TTL threshold (items with TTL <= this go to TEMP)
-o temporary_ttl=61
```

**Eviction order:**
1. Expired items (reclaimed lazily on access or by LRU crawler)
2. COLD queue tail (least recently used items)
3. If all items in a slab class are active, eviction still happens from COLD tail

### Consistent Hashing

Memcached itself has no built-in clustering. Distribution is purely client-side:

**Ketama consistent hashing (standard algorithm):**
- Each server is mapped to ~100-200 points on a hash ring (virtual nodes)
- Keys are hashed to the ring; the nearest server clockwise owns the key
- When a server is added/removed, only ~1/N of keys are remapped (N = number of servers)
- Without consistent hashing, a simple modulo (key % N) remaps nearly all keys on topology change

**Client-side configuration (libmemcached example):**
```c
memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_DISTRIBUTION,
                       MEMCACHED_DISTRIBUTION_CONSISTENT_KETAMA);
memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED, 1);
```

**Proxy-based distribution (mcrouter, twemproxy):**
- Consistent hashing is handled by the proxy
- Applications connect to the proxy as if it were a single Memcached instance
- Proxy routes commands to the correct backend server
- Benefits: language-agnostic hashing, connection pooling, failover

### Multi-Threaded Architecture

Unlike Redis (single-threaded command execution), Memcached is multi-threaded from the ground up:

- **Listener thread**: Accepts new connections, distributes to worker threads round-robin
- **Worker threads** (`-t N`, default 4): Each runs its own event loop (libevent-based), handles commands for assigned connections
- **LRU maintainer thread**: Background LRU management and slab rebalancing
- **LRU crawler thread**: Background expired-item reclamation
- **Hash table expansion thread**: Resizes the hash table without blocking workers
- **Slab rebalance thread**: Moves pages between slab classes

**Thread safety:**
- Global hash table protected by per-bucket locks (fine-grained locking)
- Slab class operations use per-class locks
- Item-level operations use per-item reference counting and CAS for consistency
- Connection state is per-thread (no sharing)

**Tuning thread count:**
```bash
# Set to number of CPU cores (hyperthreading cores count)
-t 8
# Monitor per-thread stats
stats conns    # shows connections per thread
```

### Cache Strategies

Memcached supports several caching patterns at the application level:

**Cache-Aside (Lazy Loading) -- most common:**
```
1. Application: GET key from Memcached
2. Cache miss: query database, compute result
3. Application: SET key result EX ttl
4. Next request: cache hit (fast path)
```

**Write-Through:**
```
1. Application: write to database
2. Application: SET key new_value in Memcached
3. Reads always find fresh data in cache
4. Downside: every write hits both database and cache
```

**Write-Behind (Write-Back):**
```
1. Application: SET key new_value in Memcached
2. Asynchronous process: flush dirty entries to database
3. Fastest writes but risk data loss (Memcached is volatile)
4. Rarely used with Memcached due to no persistence
```

**Cache Stampede Prevention:**
```
# Problem: popular key expires, 100 threads all query the database
# Solution 1: Locking (use ADD as a lock)
ADD lock:key 1 EX 10 NR        # Only one thread wins
# Winner: fetches from DB, sets cache, deletes lock
# Losers: retry after short sleep

# Solution 2: Early recomputation
# Store TTL metadata in the value; recompute before actual expiry

# Solution 3: Stale-while-revalidate
# Serve stale value while one thread refreshes in the background
```

### Protocol: Text vs Binary

Memcached supports two wire protocols:

**Text protocol (ASCII):**
```
# Simple, human-readable, telnet-friendly
set mykey 0 3600 5\r\n
hello\r\n
STORED\r\n

get mykey\r\n
VALUE mykey 0 5\r\n
hello\r\n
END\r\n
```

**Binary protocol:**
- Fixed-size header (24 bytes request, 24 bytes response)
- More efficient parsing (no text scanning)
- Supports silent/quiet mutations (no response unless error)
- Required for SASL authentication
- Better pipelining with opcodes and sequence numbers

**Meta commands protocol (1.6.x):**
Memcached 1.6.x introduced meta commands (`mg`, `ms`, `md`, `mn`, `me`, `ma`) that replace and extend both text and binary protocol commands:

```
# Meta get with flags
mg mykey t v f     # t=TTL remaining, v=value, f=client flags
# Response: VA 5 t=3200 f0\r\nhello\r\n

# Meta set with options
ms mykey 5 T3600 F0\r\nhello\r\n
# T=TTL, F=client flags

# Meta delete
md mykey

# Meta arithmetic
ma mykey D+       # increment; D- for decrement

# Meta noop (pipeline flush)
mn
```

Meta commands provide richer semantics: win-flag for race prevention, CAS tokens inline, opaque tokens for pipelining, and stale-while-revalidate support.

### SASL Authentication

Memcached supports SASL (Simple Authentication and Security Layer) over the binary protocol:

```bash
# Enable SASL authentication at startup
-S                        # Enable SASL
-o sasl_env_file=/path/to/envfile  # Specify credentials file (1.6.x+)
```

**SASL mechanisms supported:** PLAIN (username/password over binary protocol). Requires clients to use the binary protocol.

**Limitations:**
- No TLS built-in (use stunnel, sidecar proxy, or VPC networking for encryption)
- No per-key ACLs (authenticated or not, full access)
- AWS ElastiCache Memcached does not support SASL; it relies on VPC security groups

### Memcached vs Redis -- Decision Criteria

| Factor | Memcached | Redis |
|---|---|---|
| **Data model** | Simple key-value (string blobs) | Rich data structures (strings, hashes, lists, sets, sorted sets, streams, etc.) |
| **Threading** | Multi-threaded (scales with cores) | Single-threaded command execution (I/O threads in 6.0+) |
| **Memory efficiency** | Slab allocator, predictable overhead | jemalloc, encoding-dependent overhead |
| **Persistence** | None (pure cache) | RDB, AOF, hybrid |
| **Max item size** | 1 MB default (configurable to 128 MB) | 512 MB |
| **Clustering** | Client-side or proxy (mcrouter/twemproxy) | Built-in Redis Cluster (hash slots) |
| **Pub/Sub** | No | Yes (channels, patterns, streams) |
| **Scripting** | No | Lua scripting, Functions (7.0+) |
| **Eviction** | Segmented LRU per slab class | Configurable (LRU, LFU, volatile-*, noeviction) |
| **Replication** | No built-in (proxy layer or app-managed) | Built-in async replication |
| **Protocol** | Text, binary, meta commands | RESP2/RESP3 |
| **Best for** | Simple, high-throughput caching with multi-core scaling | Feature-rich caching, data structures, messaging, primary data store |

**Choose Memcached when:**
- Workload is simple GET/SET with string values
- Need to scale linearly with CPU cores on a single node
- Want the simplest possible caching layer
- Memory efficiency for small objects is critical (slab allocator wastes less than jemalloc for uniform sizes)
- No persistence requirements

**Choose Redis when:**
- Need data structures (hashes, sorted sets, lists)
- Need persistence or replication
- Need pub/sub or streams
- Need atomic operations beyond CAS (Lua scripting)
- Need built-in clustering

### Proxy Layer: mcrouter and twemproxy

**mcrouter (Facebook/Meta):**
- Full Memcached protocol proxy
- Consistent hashing, replication, failover, warm-up, shadowing
- Connection pooling (reduces connections to backend servers)
- Prefix-based routing (different pools for different key namespaces)
- Cold cache warm-up (reads from old pool, writes to new pool during migration)
- Used at massive scale (trillions of requests/day at Meta)

```
# mcrouter configuration example
{
  "pools": {
    "A": {
      "servers": [
        "memcached1:11211",
        "memcached2:11211",
        "memcached3:11211"
      ]
    }
  },
  "route": {
    "type": "PoolRoute",
    "pool": "A",
    "hash": {
      "hash_func": "Ch3"
    }
  }
}
```

**twemproxy (Twitter/nutcracker):**
- Lightweight proxy for Memcached and Redis
- Consistent hashing (ketama) and modula distribution
- Connection multiplexing (many client connections -> few server connections)
- Automatic server ejection and re-injection on failure
- pipelining support for batching requests
- Simpler configuration than mcrouter; fewer features

```yaml
# twemproxy (nutcracker) configuration
memcache_pool:
  listen: 0.0.0.0:11211
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  server_retry_timeout: 2000
  server_failure_limit: 3
  timeout: 400
  servers:
    - 10.0.0.1:11211:1
    - 10.0.0.2:11211:1
    - 10.0.0.3:11211:1
```

### AWS ElastiCache for Memcached

**Node types and sizing:**
- cache.t4g.* (burstable, dev/test), cache.m7g.* (general purpose), cache.r7g.* (memory-optimized)
- Memory available for caching = node memory minus OS/Memcached overhead (~10-15%)
- Up to 300 nodes per cluster (across up to 20 shards for cluster mode)

**Auto Discovery:**
- ElastiCache provides a **configuration endpoint** that returns the list of all cache nodes
- Client libraries (e.g., `elasticache-cluster-config-net`, PHP auto-discovery) query this endpoint
- When nodes are added/removed, clients automatically discover the new topology
- Polling interval configurable; default varies by client library

**Key ElastiCache considerations:**
- No SASL authentication (rely on VPC security groups and subnet groups)
- Parameter groups control Memcached configuration (chunk_size_growth_factor, max_item_size, etc.)
- Maintenance windows for patching (choose low-traffic periods)
- CloudWatch metrics: CPUUtilization, SwapUsage, Evictions, CurrConnections, NewConnections, BytesUsedForCache
- Scaling: vertical (change node type) or horizontal (add nodes, requires client rehashing)

### Client Libraries

**libmemcached (C/C++):**
- Most widely used C client library
- Supports consistent hashing, binary/text protocol, SASL, async I/O
- Language bindings: PHP (php-memcached), Python (pylibmc), Perl, Ruby

**Popular language clients:**

| Language | Library | Protocol | Consistent Hashing |
|---|---|---|---|
| Python | `pymemcache` | Text, meta | Yes (via hashring) |
| Python | `pylibmc` (wraps libmemcached) | Binary, text | Yes (built-in) |
| Java | `spymemcached` | Binary, text | Yes (ketama) |
| Java | `Xmemcached` | Binary, text | Yes (ketama) |
| Node.js | `memcached` (npm) | Text | Yes (ketama) |
| PHP | `php-memcached` (ext) | Binary, text | Yes (libmemcached) |
| Go | `bradfitz/gomemcache` | Text | Yes |
| .NET | `EnyimMemcached` | Binary | Yes (ketama) |

### Monitoring and Stats Overview

Memcached exposes rich statistics via the `stats` protocol command:

```bash
# Core stats categories
stats                  # General server statistics
stats items            # Per-slab-class item statistics
stats slabs            # Per-slab-class memory statistics
stats sizes            # Item size distribution (locks cache briefly; use stats sizes_enable)
stats cachedump <class> <count>  # Dump keys from a slab class (debugging only)
stats settings         # Server configuration parameters
stats conns            # Per-connection details
stats detail on|off|dump  # Per-prefix (namespace) statistics
```

**Key metrics to monitor:**
| Metric | Source | Alert Threshold |
|---|---|---|
| `get_hits` / (`get_hits` + `get_misses`) | `stats` | Hit ratio < 80% (investigate) |
| `evictions` | `stats` | Non-zero when unexpected (memory pressure) |
| `curr_connections` | `stats` | Approaching `-c` maxconns limit |
| `bytes` / max memory | `stats` | > 90% of `-m` allocation |
| `cmd_get` / `cmd_set` | `stats` | Read/write ratio changes |
| `incr_misses`, `decr_misses` | `stats` | Counter keys expiring unexpectedly |
| `listen_disabled_num` | `stats` | > 0 means maxconns hit |
| `evicted_unfetched` | `stats items` | Items evicted before being read even once |

For deep diagnostic commands, load `references/diagnostics.md`.

## Common Pitfalls

1. **Treating Memcached as persistent storage.** Memcached is a cache. Any restart, eviction, or node failure means data loss. Always have a backing data store and handle cache misses gracefully.

2. **Ignoring slab calcification.** Without `slab_reassign` and `slab_automove`, slab classes become imbalanced over time. This leads to evictions in one class while another has free space. Always enable slab rebalancing in production.

3. **Not using consistent hashing.** Simple modulo distribution (key % N) remaps nearly all keys when a server is added or removed. Always use consistent hashing (ketama) for production deployments.

4. **Storing items larger than the chunk size.** Items that do not fit in any slab class are rejected. If you need items > 1 MB, increase `-I` (max item size). But very large items reduce cache efficiency -- consider splitting them.

5. **Cache stampede on popular keys.** When a hot key expires, many threads simultaneously miss and hit the database. Use locking (ADD as mutex), early recomputation, or stale-while-revalidate patterns.

6. **Not monitoring eviction rates.** Evictions mean you are losing cached data before it expires. This degrades hit ratio and increases database load. Either add memory, add nodes, or reduce the dataset.

7. **Running without connection limits.** The default maxconns (`-c 1024`) may be too low for production. Each connection consumes memory for its buffer. Set `-c` appropriately and monitor `listen_disabled_num`.

8. **Ignoring multi-get hole.** When using multi-get across servers, a down server causes a "hole" -- those keys all miss, and the database sees a sudden spike of queries. Implement local fallback caching or handle partial failures gracefully.

9. **Using Memcached for data that requires atomicity beyond CAS.** Memcached has CAS (check-and-set) but no transactions, no Lua scripting, no WATCH/MULTI. If you need atomic compound operations, use Redis.

10. **Not enabling the meta protocol on 1.6.x.** Meta commands (`mg`, `ms`, `md`) provide significant advantages over the legacy text protocol: stale-while-revalidate, win flags, inline CAS, and richer responses. Upgrade client libraries to support meta commands.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Slab allocator internals, hash table, LRU segmentation, threading model, memory layout, protocol internals, meta commands. Read for "how does Memcached work internally" questions.
- `references/diagnostics.md` -- stats commands (100+ diagnostic commands), telnet/nc probes, memcached-tool usage, Prometheus exporters, hit ratio calculations, slab analysis, eviction investigation. Read when troubleshooting performance or operational issues.
- `references/best-practices.md` -- Startup flag tuning, memory sizing, consistent hashing setup, cache strategy patterns, mcrouter/twemproxy configuration, ElastiCache operational guidance, security, monitoring. Read for configuration and operational guidance.