Scaling and Traffic Distribution
Learn the patterns engineers use to handle growth — load balancing, horizontal scaling, and caching.
When a system grows, the question is never if you will need to scale — it is which part will need to scale first. This lesson covers the core patterns.
Scaling helps a system handle changing demands. Two common approaches are vertical scaling and horizontal scaling.
Vertical vs. horizontal scaling
Vertical scaling means giving a single server more resources: more CPU, more RAM, faster disks. It is simple but has hard limits — there is only so large a machine you can buy, and a single machine is a single point of failure.
Horizontal scaling means adding more servers and distributing load across them. It has no theoretical upper bound and provides redundancy. It does require coordination: you need something to decide which server handles each request.
Load balancers
A load balancer sits in front of your application servers and distributes incoming requests. Common strategies include:
- Round robin — requests are distributed to each server in turn.
- Least connections — the request goes to the server with the fewest active connections.
The load balancer also performs health checks and stops sending traffic to servers that fail.
Caching
Not all requests need to touch your database. A cache stores the result of an expensive computation or query so that subsequent requests can be served from fast in-memory storage.
Key caching concepts:
| Concept | What it means |
|---|---|
| Cache hit | The requested data is found in the cache. Fast. |
| Cache miss | The data is not in the cache; the system falls back to the database. Slow. |
| TTL (time to live) | How long a cached value is considered valid before it must be refreshed. |
| Cache invalidation | The process of removing stale data from the cache when the underlying data changes. |
Caches are typically placed close to the application layer (e.g., Redis or Memcached) and occasionally at the edge via a CDN for static assets.
Key takeaways
- Prefer horizontal scaling for flexibility.
- A load balancer distributes traffic and routes around failures.
- Caching reduces load on your database and improves response times — but cache invalidation is hard, so be deliberate about what you cache.