Horizontal scaling is adding more instances of the same service, not bigger instances. Stateless services scale horizontally with no effort; stateful services need a strategy first (replication, sharding, or moving the state out of the service).
Table of contents
- Horizontal vs vertical
- The stateless requirement
- The shared state problem
- The database bottleneck
- When horizontal scaling doesn’t help
- The autoscaling patterns
- The database scaling patterns
- FAQ
Horizontal vs vertical
The two ways to handle more load:
- Vertical scaling. Make the instance bigger. More CPU, more RAM, more disk. The instance stays at 1; it’s just a more powerful instance.
- Horizontal scaling. Add more instances. Each instance is the same size; you have more of them.
Vertical scaling has hard limits (the biggest instance the cloud offers) and is risky (changing instance size usually means downtime). Horizontal scaling has nearly infinite headroom and is the modern pattern.
The stateless requirement
Horizontal scaling requires the service to be stateless. A stateless service:
- Holds no session state in memory. The session is in a cookie, a token, or a shared store.
- Reads and writes to a shared database. All instances see the same data.
- Can be killed and replaced at any time. No graceful shutdown required.
The team that has a stateless service can scale to 1000 instances with no code changes. The team that has a stateful service has to refactor before horizontal scaling.
The shared state problem
Three patterns for handling state:
- Move state to a database. The service reads/writes Postgres, Redis, or similar. All instances share the same state.
- Move state to a client-side token. JWT in a cookie, OAuth access token in a header. The service is stateless because the state is in the token.
- Sticky sessions. The load balancer routes the same user to the same instance. Workable but breaks horizontal scaling (you can’t scale down without losing sessions).
The team that wants clean horizontal scaling picks the first two. Sticky sessions are a last resort.
The database bottleneck
Once the application scales horizontally, the database becomes the bottleneck. The patterns:
- Vertical scaling the database. Bigger instance, more RAM, faster disk. Vertical scaling is fine for the database (most databases don’t horizontal-scale cleanly).
- Read replicas. Reads go to replicas, writes go to the primary. The team that has more reads than writes (most apps) scales reads horizontally.
- Caching. Redis or Memcached in front of the database. The team that caches the hot queries has a 10x reduction in database load.
- Sharding. Split the data across multiple database instances by some key (user ID, tenant ID, region). The right pattern for very large datasets.
The team that has the application horizontally scaled and the database vertically scaled is in the right place for most workloads.
When horizontal scaling doesn’t help
The three cases where horizontal scaling is not the answer:
- Single-threaded CPU-bound work. A workload that uses 100% of one core and can’t use more cores needs vertical scaling (bigger CPU) or a different algorithm.
- Stateful protocols. Long-lived connections with state (WebSockets, game sessions) are hard to scale horizontally. The team that uses sticky sessions or a stateful broker (Redis, etc.) handles this.
- External API rate limits. If the bottleneck is an external API’s rate limit, more instances don’t help. The team needs to negotiate higher limits or batch requests.
The autoscaling patterns
The right way to autoscale horizontally:
Reactive (CPU-based). Add instances when CPU > 70%; remove when CPU < 30%. The right choice for CPU-bound workloads.
Reactive (queue-based). Add instances when queue depth > N; remove when depth < M. The right choice for worker pools.
Predictive (schedule-based). Add instances at known peak times; remove at off-peak. The right choice for predictable traffic patterns (e-commerce weekends, B2B business hours).
Predictive (ML-based). Use historical patterns to predict future load; scale ahead. The right choice for highly variable workloads.
The team that picks the right pattern has the right balance of cost and responsiveness. The team that uses reactive-only for spiky workloads has late scale-up and over-provisioned steady-state.
The database scaling patterns
Horizontal scaling at the database tier is harder. The patterns:
Read replicas. Asynchronous copies of the primary. Reads go to replicas; writes go to primary. The right choice for read-heavy workloads.
Sharding. Split data across multiple database instances by some key (user ID, tenant ID, geography). The right choice for very large datasets.
NewSQL. Distributed SQL databases (CockroachDB, Yugabyte, Spanner) that handle horizontal scaling internally. The right choice for new applications that need global scale.
CQRS. Separate read and write models. Writes go to a normalized store; reads go to a denormalized store optimized for queries. The right choice for complex domains with different read and write patterns.
Caching. Cache the hot queries in Redis or Memcached. The right choice when 80% of reads hit 20% of data (the typical 80/20 rule).
The team that picks the right database scaling pattern has the right balance of performance, cost, and complexity. The team that uses sharding for a 10 GB database is over-engineering.
FAQ
What’s the difference between horizontal and vertical scaling?
Vertical is bigger instances. Horizontal is more instances. Horizontal has nearly infinite headroom; vertical has hard limits.
When should I use horizontal scaling?
Almost always. It’s the modern pattern. The team that uses horizontal scaling by default and vertical scaling for the database is in the right place.
Can I scale a stateful service horizontally?
With effort. The state needs to move out of the service (to a database, a cache, or a token), or the instances need to be coordinated (sticky sessions, shared memory). The team that wants clean horizontal scaling makes the service stateless first.
How do I know when to scale?
Set up autoscaling based on CPU or request rate. The team that uses autoscaling scales up before the user notices; the team that scales manually is always late.
What’s the best autoscaling metric?
Depends on the workload. CPU for CPU-bound, queue depth for worker pools, request rate for web apps, custom metric for specialized workloads. The team that uses the right metric scales appropriately.
Can I autoscale a stateful service?
With difficulty. Stateful services (databases, message brokers) typically need fixed-size clusters. The team that needs to scale a stateful service uses cluster scaling (add a node to the cluster) rather than instance scaling.
How do I know if my service is ready to scale horizontally?
The team that has stateless application servers, no session affinity, and a shared database has a service ready to scale horizontally. The team that has sticky sessions, in-memory caches, or local file storage is not ready.
If you are sizing the infrastructure for the kind of project this post covers, the RunxBuild hosting calculator is the right place to model the line items. The compute, the memory, the storage, the bandwidth, the database - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers. The RunxBuild dashboard is where the team sees the actual usage in one place.
Useful related references: