Scaling an application means adding capacity to handle more load without the response time or the error rate going up. There are two ways to add capacity: vertical (a bigger box) and horizontal (more boxes). That is the short version. The longer version is that the “horizontal vs vertical” question is the wrong starting point. The right starting point is the bottleneck. Different bottlenecks want different answers.
Most “scaling” posts skip the bottleneck question and jump to the framework. The framework is useful, but it is the second question, not the first. The first question is: where is the app slow, and what does the load look like? The answer to that question picks the scaling strategy. The answer to “horizontal or vertical” is almost always “both, in different layers.”
This post is a working engineer’s take on scaling an application: how to find the bottleneck, when vertical wins, when horizontal wins, when the database is the actual constraint, and when autoscaling is the right answer. It assumes you are running a real app in production, that you have hit at least one “we need to scale” moment, and that you want a decision tree you can actually use.
Table of contents
- The short version
- The question before the question
- The three bottlenecks that decide the strategy
- Vertical scaling: when a bigger box is the right answer
- Horizontal scaling: when more boxes is the right answer
- The database is the actual constraint
- Stateless vs stateful: the design decision that decides everything
- Autoscaling: the part that actually scales
- The cost curve that hides in every plan
- The pre-mortem: what fails first when traffic spikes
- The capacity plan that fits on one page
- FAQ
The short version
Start with the bottleneck. Profile the app under realistic load. The bottleneck is almost always the database, the network, or a single-threaded CPU path. Once you know the bottleneck, the scaling strategy is mostly determined.
Vertical scaling is faster to do and easier to understand. It is the right answer for monolithic apps with a single hot path, for stateful workloads that cannot be distributed, and for the first six months of a new product. The ceiling is real but high. Most production apps never need to leave vertical.
Horizontal scaling is more work to set up and easier to operate at scale. It is the right answer for stateless services with predictable per-request work, for apps with bursty traffic, and for any workload that has to survive a regional outage. The trade is operational complexity for elastic capacity.
The database is the actual constraint in most production systems. A 100-instance app fleet backed by a single Postgres is bottlenecked at the database, not at the app fleet. The fix is to scale the database: read replicas for read-heavy workloads, vertical scaling of the primary for write-heavy workloads, connection pooling to handle the connection fan-in.
Autoscaling is what makes either of the above actually elastic. The pattern is a metric (CPU, request rate, queue depth), a target, and a min/max range. The autoscaler adds capacity when the metric crosses the target and removes capacity when it drops below. The app is stateless, the database is durable, the autoscaler is the policy.
The question before the question
Before asking “how do I scale,” the question is “what is slow?” The answer is almost never “the whole app is slow.” The answer is usually a specific path through the app, a specific query, a specific endpoint, or a specific downstream service.
Profile the app under realistic load. A load test that exercises the endpoints you actually care about, with the request sizes and patterns you actually see, gives you a picture of where the time goes. Tools like wrk, k6, vegeta, or Locust can generate the load. The app’s metrics (response time, error rate, CPU, memory, database query time) tell you where the bottleneck is.
Look at the slow query log. If the database is the bottleneck, the slow query log tells you which query is the bottleneck. EXPLAIN ANALYZE on the slow query tells you why. A missing index, a sequential scan on a large table, a join that returns too many rows — these are the typical causes.
Look at the network. If the app is waiting on the network (time to first byte, time to last byte, DNS resolution), the network is the bottleneck. A CDN, a connection pool, a regional endpoint, or a different network path may be the answer.
Look at the CPU. If the app is CPU-bound, the code path is doing too much work per request. The fix is usually code (a faster algorithm, a smaller payload, a cache), not a bigger box. A bigger box is a temporary fix; the CPU-bound path will eventually saturate any box.
Look at the memory. If the app is memory-bound, the workload is holding too much in memory. The fix is usually code (streaming, pagination, smaller in-memory data structures), or a bigger box if the workload is legitimately large.
Look at the disk. If the app is disk-bound (high I/O wait, slow reads), the disk is the bottleneck. A faster disk (NVMe instead of SSD), more memory for caching, or a different storage layout may be the answer.
The bottleneck is almost always one of these: database, network, CPU, memory, or disk. The scaling strategy depends on which one it is.
The three bottlenecks that decide the strategy
Every scaling decision comes back to one of three bottlenecks.
Compute-bound bottlenecks are solved by adding CPU. A CPU-bound path can be vertical-scaled (bigger box) or horizontal-scaled (more boxes behind a load balancer). The choice depends on the workload.
I/O-bound bottlenecks are solved by reducing the I/O. A database-bound app is I/O-bound; the answer is to reduce the database load (caching, read replicas, query optimization) before adding more boxes. A network-bound app is I/O-bound; the answer is to reduce the network load (CDN, compression, regional endpoints) before adding more boxes.
State-bound bottlenecks are solved by rethinking the state. A stateful workload (a session, a queue, a lock) is bottlenecked at the state, not the app. The answer is to move the state to a service designed for it (Redis, Postgres, a queue), not to add more boxes of the same stateful app.
The trap is treating every bottleneck as a compute bottleneck. The team says “we need more boxes” when the database is the actual constraint. The team adds five more app instances, the database is still the bottleneck, and the bill goes up. The right answer is to find the bottleneck first, then add capacity in the layer that needs it.
Vertical scaling: when a bigger box is the right answer
Vertical scaling means a bigger instance: more CPU, more memory, faster disk. The same code, the same config, the same data. Just a bigger box.
The right answer for monolithic apps. A monolithic app with a single hot path is hard to scale horizontally. Splitting the monolith is months of work. A bigger box is hours of work. For a small team with a growing monolith, vertical is almost always the right answer.
The right answer for stateful workloads. A database, a queue, a cache — anything that holds state — is hard to scale horizontally without sharding. A bigger box is the right answer until the workload is large enough to justify the sharding complexity. Postgres on a 32-vCPU box with 128 GB of RAM is a real production setup. Most apps never exceed it.
The right answer for the first six months of a new product. New products have unknown load patterns. Horizontal scaling requires a stateless design, an autoscaler, a load balancer, and a database that can handle the connection fan-in. Vertical scaling requires a bigger box. The first six months is not the time to invest in the horizontal scaffolding.
The ceiling is real. Vertical scaling has a hard ceiling. The biggest instance on most clouds is 192 vCPU, 768 GB of RAM, or similar. Beyond that, horizontal scaling is the only option. The ceiling is high enough that most apps never hit it, but it is real.
The cost curve is steep at the top. Vertical scaling costs more per unit of capacity at the top of the range. A 64-vCPU box is more than 2x the price of a 32-vCPU box. The biggest instance is more than 4x the price of a mid-range one. The diminishing return is real, and the team should know it before committing to vertical.
The blast radius is bigger. A bigger box is a single point of failure. If the box dies, the workload dies. The mitigation is a multi-AZ setup with a hot standby, but the standby is another big box that costs the same.
Horizontal scaling: when more boxes is the right answer
Horizontal scaling means more instances of the same app, behind a load balancer. The app is stateless, the load balancer distributes the requests, and the database is shared.
The right answer for stateless services. A stateless service (a REST API, a static site, a worker that does not hold state between requests) is the canonical horizontal-scaling target. Add a box, the load balancer sends some traffic to it, the new box serves the requests, the box is removed when the load drops.
The right answer for bursty traffic. A workload with predictable peaks (a news site during a story, a ticketing app during an event, a SaaS during a month-end) is a good horizontal-scaling target. The autoscaler adds boxes when the load increases, removes them when it drops. The cost is proportional to the actual load.
The right answer for high availability. A horizontal fleet with three or more instances is naturally resilient to single-instance failure. The load balancer routes around the failed instance. The user sees a brief blip, not an outage. The pattern is the standard for any service-level-availability commitment.
The right answer for global deployments. A horizontal fleet can be deployed across regions. The load balancer routes to the nearest region. The user gets a low-latency response. The pattern is the standard for any global product.
The trade is operational complexity. A horizontal fleet requires a load balancer, an autoscaler, a stateless design, a shared database, a connection pool, a health check, a deployment story that rolls the fleet, and a monitoring system that can handle the fan-in. The complexity is real and the team should know it before committing to horizontal.
The trade is the database. A horizontal fleet of stateless app instances is bottlenecked at the database. The fix is to scale the database: read replicas for read-heavy workloads, connection pooling for the connection fan-in, vertical scaling of the primary for write-heavy workloads. The database is the constraint that horizontal scaling does not solve by itself.
The database is the actual constraint
In most production systems, the database is the bottleneck. A 100-instance app fleet backed by a single Postgres is bottlenecked at the database. The 100 instances are spending most of their time waiting for the database to respond.
Read-heavy workloads. The pattern is read replicas. The primary handles writes, the replicas handle reads. The app routes reads to a replica, writes to the primary. The replication lag is usually milliseconds; the app has to handle stale reads. The pattern scales reads almost linearly with the number of replicas.
Write-heavy workloads. The pattern is vertical scaling of the primary. Postgres on a 32-vCPU box with NVMe drives and a large connection pool handles a lot of writes. The ceiling is real, and the path beyond the ceiling is sharding (Citus, Vitess, or hand-rolled), but most workloads never exceed a single big Postgres.
Mixed workloads. The pattern is a combination. Reads go to replicas. Writes go to the primary. Heavy aggregation queries go to a separate analytics database (a data warehouse, a read-only Postgres snapshot). The app routes by query type.
Connection fan-in. A horizontal fleet of 100 app instances, each opening 10 connections to Postgres, is 1,000 connections. Postgres handles around 100 to 500 concurrent connections by default. The fix is a connection pool (PgBouncer, RDS Proxy, or the platform’s managed database). The pool reduces the connection count and adds a queue.
Query optimization. The cheapest scaling is the query that does not have to run. A missing index, a sequential scan on a large table, a join that returns too many rows — these are the typical causes. The fix is in the database, not in the fleet.
The right answer is almost always “scale the database” before “scale the fleet.” The fleet is the visible part. The database is the actual constraint.
Stateless vs stateful: the design decision that decides everything
The single most important design decision for scaling is whether the app is stateless or stateful. A stateless app is horizontally scalable. A stateful app is not.
Stateless means no in-memory state that has to survive a request. The app reads from the database, computes, returns. The next request is the same. There is no “logged-in user” stored in the app’s memory; the session is in a cookie or a session store. There is no “open file” stored in the app’s memory; the file is in an object store.
Stateful means there is in-memory state that has to survive a request. The app holds a session, a websocket connection, a long-running computation, or a lock. The state is on the app instance. A new instance does not have the state.
The pattern is to move the state out of the app. Sessions go to Redis or a database. Websockets go to a pub/sub system. Long-running computations go to a queue with a separate worker. Locks go to a distributed lock service (Redis, ZooKeeper, or a database advisory lock).
The pattern is well-known and almost always the right answer. A stateless app is easy to scale, easy to deploy, easy to roll back, and easy to debug. A stateful app is hard to scale, hard to deploy, and hard to debug. The design decision is “where does the state live?” The right answer is “somewhere that is not the app instance.”
For a RunxBuild service, the stateless pattern is the default. The service is a container, the state is in a managed database or a managed Redis, and the scaling is horizontal by default. The platform configuration handles the load balancer, the autoscaler, the health check, and the deploy story.
Autoscaling: the part that actually scales
Vertical and horizontal are the capacity options. Autoscaling is the policy that adds and removes capacity as the load changes.
The pattern is a metric, a target, and a range. A metric (CPU, request rate, queue depth, response time) is observed. A target is set (e.g., 70% CPU). A range is set (e.g., min 2 instances, max 20 instances). The autoscaler adds capacity when the metric exceeds the target, removes capacity when it drops below.
The metric is the design decision. CPU is easy to measure but lags the actual load. Request rate is more responsive but harder to correlate with capacity. Queue depth is the most responsive but requires a queue. Response time is the most user-facing but is also the most lagging.
The target is the tuning decision. A high target (e.g., 80% CPU) uses fewer instances but degrades user experience at peak. A low target (e.g., 30% CPU) uses more instances but degrades gracefully. The right target depends on the workload and the budget.
The range is the safety decision. A wide range (min 2, max 100) gives the autoscaler room to scale up, but the cost can spike if the load stays high. A narrow range (min 5, max 10) caps the cost but can saturate at peak.
The cooldown is the stability decision. The autoscaler should not add and remove capacity in rapid succession. A cooldown of 5 minutes between scale-up events and 10 minutes between scale-down events prevents flapping.
The pattern that works. A stateless service, a metric that correlates with user load, a target that leaves headroom, a range that covers the worst-case, a cooldown that prevents flapping. The autoscaler does the rest.
On a PaaS like RunxBuild, the autoscaler is built in. The configuration is the metric, the target, the range, and the cooldown. The platform handles the rest.
The cost curve that hides in every plan
The cost curve for scaling is not linear. The team that assumes “twice the load means twice the cost” is going to be surprised.
Vertical scaling has a step function. The cost is the cost of the next instance size up. The next size is usually 2x the current size. The cost is 2x for 2x the capacity, which sounds like a deal until the team realizes they needed 1.5x and not 2x.
Horizontal scaling has a different curve. The cost is the cost of each new instance, which is the same per unit. The cost scales linearly with the number of instances. The trade is that the team can scale up by 1.1x by adding one instance, and the cost is 1.1x.
The database is the expensive part. A 10-instance app fleet backed by a single Postgres is not 10x the cost of a 1-instance app. The database is a single instance. The cost is the app fleet (linear) plus the database (which may be a single big instance, or a primary plus replicas). The database can be the majority of the cost for a stateful workload.
The calculator is the answer. The honest exercise is to run the numbers for the workload at the current load, at 2x the load, at 5x the load, and at 10x the load. The cost curve is rarely linear. The team that knows the curve makes better decisions.
The cost that hides is the database. The team that plans for 100 app instances and forgets to plan for the database is going to find that the database is the constraint at 50 instances. The right answer is to plan the database capacity alongside the app capacity, with a clear understanding of which layer the autoscaler is watching.
The pre-mortem: what fails first when traffic spikes
A pre-mortem is the exercise of imagining the failure before it happens. The team asks: “It is 90 days from now, and the scaling plan failed. What went wrong?” The answers are the risks to plan for.
The database saturated first. The most common failure. The app fleet scaled to 100 instances, the database is still a single instance, the database saturated at 50 instances, the app is waiting on the database, the response time went up, the error rate went up. The fix is to plan the database capacity with the app capacity.
The autoscaler did not scale fast enough. The traffic spike was faster than the autoscaler could react. The cooldown was too long, the metric was lagging, the new instance took 5 minutes to start. The fix is to set the cooldown to a value that matches the rate of change of the load, and to over-provision for the first minute of a spike.
The connection pool saturated. The app fleet scaled to 100 instances, the connection pool was sized for 10, the pool exhausted, the new instances could not connect to the database, the error rate went up. The fix is to size the connection pool for the maximum fleet size, not the current fleet size.
The health check failed. The new instance took longer to start than the health check allowed. The autoscaler marked the instance as unhealthy, removed it, and tried again. The fleet oscillated. The fix is to set the health check timeout to a value that matches the actual startup time, and to set the cooldown to a value that allows the instance to settle.
The deploy story failed. A horizontal fleet has a deploy story that rolls the fleet. If the deploy story is “replace all instances at once,” the fleet is unavailable for the duration of the deploy. The fix is a rolling deploy, a canary deploy, or a blue-green deploy.
The network saturated. The app fleet scaled to 100 instances, the network between the fleet and the database saturated, the response time went up. The fix is a private network with enough bandwidth, or a regional deployment that keeps the fleet and the database close.
The pre-mortem is the cheapest insurance. The team that runs it before the spike saves the postmortem after the spike.
The capacity plan that fits on one page
A useful capacity plan is a single page with the following sections.
The workload description. What does the app do, what is the request rate, what is the average response time, what is the peak request rate, what is the peak response time, what is the error rate at peak.
The bottleneck analysis. What is the bottleneck under current load, what is the bottleneck at 2x the load, what is the bottleneck at 5x the load, what is the bottleneck at 10x the load.
The scaling strategy. Vertical, horizontal, or both. For each layer (app, database, cache, queue), the strategy and the trigger.
The capacity numbers. Current capacity, capacity at 2x, capacity at 5x, capacity at 10x. The numbers are instances, CPU, memory, disk, network, and the cost in the calculator.
The pre-mortem. The top three failure modes and the mitigation for each.
The runbook. The steps to scale up, the steps to scale down, the steps to roll back a bad scale.
The plan fits on one page. The plan is reviewed quarterly. The plan is updated when the workload changes. The plan is the answer to “what do we do when the load doubles.”
FAQ
What is the difference between horizontal and vertical scaling?
Vertical scaling is a bigger box: more CPU, more memory, faster disk, on the same instance. Horizontal scaling is more boxes: more instances of the same app, behind a load balancer. Vertical is faster to set up; horizontal scales further.
When should I scale vertically?
Scale vertically for monolithic apps, stateful workloads, and the first six months of a new product. Vertical is the right answer when the bottleneck is a single hot path that is hard to distribute. The ceiling is real but high; most apps never need to leave vertical.
When should I scale horizontally?
Scale horizontally for stateless services, bursty traffic, high availability, and global deployments. Horizontal is the right answer when the workload can be distributed and the team has the operational capacity to manage a fleet. The trade is operational complexity for elastic capacity.
What is the database scaling strategy?
Read replicas for read-heavy workloads. Vertical scaling of the primary for write-heavy workloads. Connection pooling for the connection fan-in. Query optimization for the queries that do not have to run. A 100-instance app fleet backed by a single Postgres is bottlenecked at the database.
How does autoscaling work?
A metric (CPU, request rate, queue depth, response time) is observed. A target is set. A range is set (min and max instances). A cooldown is set. The autoscaler adds capacity when the metric exceeds the target, removes capacity when it drops below. The pattern is stateless service, responsive metric, target with headroom, range that covers the worst case.
What is the cost of scaling?
The cost depends on the strategy. Vertical scaling has a step cost (the next instance size). Horizontal scaling has a linear cost (one instance at a time). The database is the expensive part for stateful workloads. The honest answer is in the calculator, not in a back-of-the-envelope estimate.
What is the difference between stateless and stateful?
Stateless means no in-memory state that has to survive a request. The app reads from the database, computes, returns. Stateful means there is in-memory state (a session, a websocket, a long-running computation) that has to survive. Stateless apps are easy to scale horizontally. Stateful apps are not.
How do I know if my app is stateless?
Look at the in-memory state. If the app holds anything between requests (a session, a websocket, a lock, an open file), it is stateful. If the app only holds the request in memory and returns, it is stateless. The fix for a stateful app is to move the state to a service (Redis, a database, a queue).
What is the most common scaling mistake?
Treating the fleet as the bottleneck when the database is. The team adds 10 more app instances, the database is still a single instance, the database saturates, the response time goes up. The right answer is to find the bottleneck first, then add capacity in the layer that needs it.
How do I know when to scale?
The honest answer is “when the metrics say so.” Set a target (70% CPU, 50 ms response time, 100 requests per second per instance). When the metric exceeds the target, scale. When the metric drops below, scale down. The pre-mortem and the runbook are the safety net for the cases the metric does not catch.
What is the difference between scaling and autoscaling?
Scaling is the act of adding capacity. Autoscaling is the policy that does it automatically. A scaling strategy without an autoscaler requires a human to add capacity when the load increases. A scaling strategy with an autoscaler is the production-ready version.