Error 503 Backend Is Unhealthy: The Health Check Is Failing, Not the Site

This error comes from a proxy or load balancer in front of your application, not from the application itself. It means the health check the proxy runs against your backend has been failing, so the proxy has stopped sending traffic there. Your application may be perfectly healthy and failing the check for an unrelated reason, and distinguishing those two cases is the whole job.

The wording is misleading in a specific way. Unhealthy is the proxy’s conclusion, not a measurement of your service. A backend that responds correctly to real requests can be marked unhealthy because the check asks for a path that does not exist, and that scenario is more common than an actual outage.

Table of contents

• What is actually happening
• Check from where the proxy stands
• Health checks that are worth having
• Varnish, Fastly, and stale content
• When it is genuinely your application
• How this fits the rest of the stack
• FAQ

What is actually happening

A proxy configured with health checking polls each backend on an interval, requesting a specific path and expecting a specific response. When a number of consecutive checks fail, the backend is marked unhealthy and removed from the pool. With no healthy backends left, the proxy has nothing to forward to and returns a 503 with this message.

So there are four independent things that can be wrong, and only one is your application being broken.

1. The application is genuinely down or crashing.
2. The application is fine but the health check is misconfigured, asking for the wrong path, port, or expecting the wrong status.
3. The application is fine but too slow to answer within the check’s timeout, usually under load.
4. The application is fine and reachable by users, but not reachable from the proxy, which is a network or firewall problem.

Cases two and four produce a site that is completely down while every diagnostic you run from your own machine says the application is healthy. That mismatch is the signature of this error, and chasing the application when you see it wastes the most time.

Check from where the proxy stands

The critical move is to run the health check yourself, from the proxy’s position, exactly as the proxy runs it.

# From the proxy or load balancer host, not your laptop.
curl -v -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' \
  http://10.0.1.42:8080/health

# Is anything listening there at all?
nc -zv 10.0.1.42 8080

# What is the backend bound to?
sudo ss -tlnp | grep 8080

Four things to compare against the configured check: the exact path, the port, the expected status code, and the time taken against the configured timeout.

The most common findings, in order. The health path returns 404 because the application never implemented it, or it moved. The check expects 200 and the endpoint returns 301 because the application redirects everything to HTTPS, including the internal check. The response takes longer than the timeout under load. Or the backend is bound to 127.0.0.1 and unreachable from another host.

That redirect case catches people repeatedly. An application that forces HTTPS will redirect the proxy’s plain HTTP health check, the proxy sees a 301 rather than the expected 200, and marks the backend unhealthy. The site works for users because their traffic follows the redirect; the health check does not follow it.

Health checks that are worth having

Most health check endpoints are either useless or actively harmful, and the difference is what they measure.

An endpoint that returns 200 unconditionally tells you the process is running and nothing else. An application that has lost its database connection passes it happily while every real request fails.

An endpoint that checks every dependency has the opposite problem. If it queries the database, calls two internal services, and pings a cache, then any one of those being briefly slow marks your backend unhealthy and removes it from the pool, turning a minor dependency blip into a full outage. Worse, if all your instances share the dependency, they all fail the check simultaneously and the whole service disappears.

The distinction that resolves this is two separate endpoints.

// Liveness: is this process functional? Cheap, no dependencies.
// Failing this should restart the instance.
app.get("/healthz", (req, res) => res.status(200).send("ok"));

// Readiness: can this instance serve traffic right now?
// Failing this should remove it from the pool, not restart it.
app.get("/readyz", async (req, res) => {
  try {
    await db.query("SELECT 1");        // own dependencies only
    res.status(200).json({ status: "ready" });
  } catch (err) {
    res.status(503).json({ status: "not ready", reason: err.code });
  }
});

The rule for readiness: check only what this instance owns and cannot function without. Do not check downstream services you merely call, because their failure should degrade your responses rather than remove you from the pool.

Keep both fast, exempt from authentication, exempt from rate limiting, and exempt from the HTTPS redirect. All four of those have caused this error in production.

Varnish, Fastly, and stale content

In a Varnish or Fastly setup, this message has a particular meaning worth knowing. Varnish tracks backend health with a probe and, when the backend is unhealthy, will serve stale cached content rather than an error if the grace period allows it.

That is a good behaviour and it means a 503 in that stack tells you two things happened: the backend failed its probe, and there was no graced content available to serve. Extending the grace period turns a class of brief backend problems into cached responses nobody notices.

backend default {
  .host = "10.0.1.42";
  .port = "8080";
  .probe = {
    .url = "/healthz";
    .timeout = 2s;
    .interval = 5s;
    .window = 10;
    .threshold = 6;    # 6 of the last 10 must pass
  }
}

The window and threshold settings are the ones to tune. A threshold that is too strict marks a backend unhealthy on a single slow response; too lenient and traffic keeps going to a broken instance. Requiring a majority of a rolling window is a reasonable default.

Also check the timeout against reality. A two-second probe timeout against an endpoint that touches a database under load will fail intermittently, and intermittent health check failures produce exactly the flapping behaviour that is hardest to diagnose.

When it is genuinely your application

Having ruled out the check itself, the usual causes are ordinary.

• The process crashed and the supervisor is restarting it in a loop. Check the service status and the restart count, not just whether it is running now.
• It ran out of memory and was killed. Look for the out-of-memory killer in the kernel log.
• It is up but saturated, with every worker busy, so new connections queue past the timeout.
• It cannot reach its own database, so it is failing readiness correctly and the health check is doing its job.
• A recent deploy is the cause, which is worth checking first if the timing lines up.

sudo systemctl status myapp
journalctl -u myapp --since "15 min ago" -p err
dmesg -T | grep -i 'killed process'
ss -s

The saturation case is the one that looks like an infrastructure problem and is not. Under enough load, response times cross the health check timeout, instances get marked unhealthy and removed, the remaining instances receive more traffic, and they cross the threshold too. The pool empties in sequence, which reads as a cascading infrastructure failure and is really a capacity problem with a health check amplifying it.

Autoscaling between a floor and a ceiling addresses that shape directly, by adding capacity before response times reach the timeout rather than after.

How this fits the rest of the stack

This error is a good argument for having the deploy log and the runtime log in the same place, because the first useful question is almost always whether anything changed just before it started. Services on RunxBuild keep build logs and runtime logs per deploy with a rollback to the previous version, and autoscaling between a floor and ceiling plan covers the saturation case. The RunxBuild hosting calculator shows the service and its database as separate line items.

Useful related references:

• HTTP 503: What Is Actually Broken When the Service Is Unavailable
• Maintenance Page: The Version That Actually Helps Users Come Back, Not a 503 Wall
• Deploy a Go Backend for Free on RunxBuild
• Services on RunxBuild

FAQ

What does 503 backend is unhealthy mean?

A proxy or load balancer in front of your application ran a health check against it, the check failed repeatedly, and the backend was removed from the pool. With no healthy backends left, the proxy returns 503.

Why is my backend unhealthy when the site works?

Most often the health check is misconfigured rather than the application being broken. Common causes are a health path that 404s, an HTTPS redirect turning the check’s 200 into a 301, a timeout shorter than the response time, or the backend bound to localhost only.

What should a health check endpoint check?

Use two. Liveness should be cheap and dependency-free, so failing it restarts the instance. Readiness should check only dependencies this instance cannot function without, so failing it removes the instance from the pool without restarting it.

Should my health check test the database?

The readiness check should, since an instance that cannot reach its database cannot serve traffic. The liveness check should not, or a brief database blip restarts every instance simultaneously.

Why do backends flap between healthy and unhealthy?

Usually a probe timeout close to the real response time under load, so checks fail intermittently. Raise the timeout, require a majority of a rolling window rather than a single failure, and check whether the instances are simply saturated.