A server that was down is rarely the server. It is usually the network, the DNS, the load balancer, the dependency, the application, or the database. The pattern that catches most “server was down” reports: the server was up, the user could not reach it, and the team concludes the server is down.
Table of contents
- Check the network
- Check the DNS
- Check the load balancer
- Check the dependency
- Check the application
- Check the database
- FAQ
Check the network
The first thing to check: is the network up? The pattern: the user cannot reach the server, the network is down, and the user concludes the server is down.
The fix: test the network. ping or traceroute from a known-good location. If the network is up, the problem is not the network.
Check the DNS
The second most common cause is DNS. The pattern: the user types a URL, the DNS resolver is down or slow, the connection fails, and the user concludes the server is down.
The fix: time the DNS resolution. dig +short on Linux or nslookup on Windows. If the resolution fails, the problem is the DNS. The fix is usually a faster DNS resolver (1.1.1.1, 8.8.8.8) or a local cache.
Check the load balancer
The third most common cause is the load balancer. The pattern: the user reaches the load balancer, the load balancer has no healthy targets, the load balancer returns 503, and the user concludes the server is down.
The fix: check the load balancer’s target group. If the target group has no healthy targets, the problem is the targets, not the load balancer.
Check the dependency
The fourth most common cause is a dependency. The pattern: the user reaches the server, the server tries to reach a dependency (a database, a queue, an external service), the dependency is down, the request fails, and the user concludes the server is down.
The fix: check the dependencies. The team’s monitoring should show whether the dependencies are up or down. If a dependency is down, the fix is in the dependency, not the server.
Check the application
The fifth most common cause is the application. The pattern: the user reaches the server, the server is up, the dependencies are up, but the application is down (a crash, a deadlock, a memory leak).
The fix: check the application’s logs. The application should have logged the crash or the error. The fix is in the application, not the server.
Check the database
The sixth most common cause is the database. The pattern: the user reaches the server, the server is up, the application is up, but the database is down (a crash, a deadlock, a slow query).
The fix: check the database’s logs. The database should have logged the crash or the error. The fix is in the database, not the server.
FAQ
Why is my server down?
Usually not the server. The most common cause is the network, then the DNS, then the load balancer, then a dependency, then the application, then the database. Check those before you open the incident.
How do I diagnose a server that is down?
Check the network. Check the DNS. Check the load balancer. Check the dependencies. Check the application. Check the database. The answer is usually in one of those six places.
What is the most common cause of a server being down?
A dependency. The database is the most common dependency that goes down. The fix is in the dependency, not the server.
How do I prevent a server from going down?
Right-size the workload, monitor the dependencies, have a runbook for the common failure modes, and test the runbook at least once a quarter.
What is the difference between a server being down and a server being slow?
A server that is down returns an error. A server that is slow returns a response, but the response is late. The fixes are different: the down server needs to be restarted; the slow server needs to be profiled.
If you are sizing a server tier for a new project, the RunxBuild hosting calculator is the place to model the line items. The instances, the storage, the dependencies, the database - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers. The RunxBuild dashboard is where the team sees the actual server health in one place.