Caching

A cache stores the result of an expensive operation so future requests can be served from fast memory rather than recomputed or re-fetched. A well-placed cache can reduce database load by orders of magnitude and cut response times from hundreds of milliseconds to single-digit milliseconds.

Cache Layers

Caches exist at every layer of the stack. Understanding which layer to target determines what you can cache and what the trade-offs are.

1. Client-side — browser cache, HTTP cache headers (Cache-Control, ETag). Zero server load for repeated requests.
2. CDN — edge caches at Points of Presence globally. Reduces latency for static and semi-static content without touching your origin.
3. Application cache — an in-process cache (in-memory map, Caffeine in Java) or a distributed cache (Redis, Memcached). Stores computed results, database query results, and session data.
4. Database query cache — some databases internally cache query results. Largely deprecated; MySQL removed its query cache in 8.0 because it was a bottleneck under concurrent writes.
5. OS / hardware — CPU cache, OS page cache. Transparent to the application; the OS automatically caches recently accessed disk pages in RAM.

In system design, "add a cache" almost always means an application-level distributed cache — specifically Redis.

Cache Strategies

1. Cache-aside (lazy loading) — the application checks the cache first. On a miss it fetches from the database, writes to the cache, and returns the result. The most common pattern. The cache is populated only on demand, so it never holds data that is never requested. Risk: cold start on cache restart; stale data until TTL expires.
2. Read-through — the cache sits in front of the database. On a miss the cache itself fetches from the database and populates itself — the application always talks to the cache only. Simplifies application code; the cache library handles population logic.
3. Write-through — every write goes to the cache and the database synchronously. The cache is always consistent with the database. Trade-off: write latency doubles; the cache fills with data that may never be read.
4. Write-back (write-behind) — writes go to the cache immediately and to the database asynchronously. Fastest writes but carries a data loss risk if the cache crashes before flushing. Used for write-heavy workloads where a small window of potential loss is acceptable.

Eviction Policies

When the cache fills up, something must be evicted to make room. The eviction policy determines what gets removed.

1. LRU (Least Recently Used) — evicts the item not accessed for the longest time. The most common policy; works well when recent access predicts future access (temporal locality).
2. LFU (Least Frequently Used) — evicts the item accessed the fewest total times. Better for skewed access patterns where a small set of hot keys are accessed very frequently. Downside: new items start with a low frequency count and are easily evicted before they get a chance to warm up.
3. FIFO (First In, First Out) — evicts the oldest inserted item regardless of access. Simple but ignores access popularity entirely.
4. TTL (Time To Live) — items expire after a fixed duration regardless of access pattern. Simple and predictable; good for data that naturally goes stale (user sessions, API responses).
5. Random replacement — evicts a random item. Surprisingly competitive with LRU in some workloads; commonly used in CPU cache design.

Cache Invalidation

Phil Karlton famously said there are only two hard problems in computer science: cache invalidation and naming things. When the underlying data changes, the cached version becomes stale.

Invalidation strategies

1. TTL expiry — accept eventual consistency up to the TTL window. No coordination needed. Choose the TTL based on how stale the data can be before it causes a user-visible problem.
2. Event-driven invalidation — when a write occurs, explicitly delete or update the relevant cache key. Requires coordination between the write path and the cache. More complex but gives near-real-time consistency.
3. Write-through — invalidation is automatic because the cache is always written on every database write. Consistency is guaranteed; the trade-off is write latency.

Cache stampede (thundering herd)

When a popular cache key expires, many requests simultaneously miss and all fire database queries at once — potentially overwhelming the database. Solutions:

1. Mutex / lock — only one request rebuilds the cache; all others wait. Reduces database pressure but adds latency for waiting requests.
2. Probabilistic early expiry — begin refreshing the cache slightly before it expires using a random jitter. Prevents a cliff-edge stampede of concurrent requests.
3. Stale-while-revalidate — serve the stale cached value immediately while asynchronously refreshing it in the background. Zero latency increase; briefly stale data.

Redis vs Memcached

Redis is almost always the right choice unless you specifically need raw multi-threaded throughput with no other features. Its rich data structure support means it doubles as a rate limiter (sorted sets), a session store (hashes), a pub/sub bus (streams), and a job queue (lists with BLPOP) — reducing the number of separate infrastructure components your system needs.