CPU usage high on Linux is debugged in three steps: confirm the issue with top or uptime, find the top consumers with ps aux --sort=-%cpu or top -o %CPU, then drill into the worst offender with perf top or strace -p <pid> for the kernel-level hot path. The team that finds the burning process fixes the bug; the team that just reboots has a recurring problem - the bug is still there.
Table of contents
- Step 1: confirm the issue
- Step 2: find the top consumers
- Step 3: drill into the process
- Common causes
- The fix: kill, restart, optimize, or scale
- How this fits the rest of the stack
- FAQ
Step 1: confirm the issue
top
uptime
uptime shows the 1, 5, and 15-minute load averages. Load average above the number of CPU cores means the system is CPU-saturated. A 4-core box with a load average of 8 has 4 processes waiting for each core - users see latency.
top shows the same plus the top processes. Press 1 to see all CPUs, P to sort by CPU, M to sort by memory. The team that monitors uses top interactively for spot-checks, then writes a script for the long-running view.
Step 2: find the top consumers
ps aux --sort=-%cpu | head -20
Output: the top 20 processes by CPU usage. The team that sees a process at 380% CPU on a 4-core box has found the offender (380% means the process is using 3.8 cores).
For a tree view (parent + children):
ps auxf --sort=-%cpu | head -30
The team that has a parent process spawning many children sees the total impact.
For an interactive tree view, htop is better than top:
sudo apt install htop
htop
F2 setup, F5 tree view, F6 sort by various columns, F9 kill. The team that uses htop has the right tool for CPU debugging.
Step 3: drill into the process
For per-thread CPU within a process:
top -H -p <pid>
Or:
ps -p <pid> -L -o pid,tid,pcpu,comm
The team that sees one thread at 99% and the rest at 1% has a single-threaded bottleneck - the process is single-threaded for the hot path.
For the kernel-level hot path:
sudo perf top -p <pid>
perf samples the process and shows the symbols (functions) consuming the most CPU. The team that has a Node.js process burning 80% CPU in [JS] or [garbage collector] has a JS-level issue; the team that sees _spin_lock_irqsave at the top has a kernel lock issue.
For syscall-level debugging:
sudo strace -p <pid> -c
After 10 seconds, Ctrl-C, and you get a syscall summary. The team that sees 90% read or write has an I/O-bound process; the team that sees 90% futex has a lock contention issue.
Common causes
-
Runaway loop - a bug in the application that has the process stuck in a tight loop. The fix: fix the bug, or restart the process temporarily.
-
Inefficient code - the process is doing too much work (N+1 queries, O(n^2) algorithms, hot inner loops). The team that has a database query running 10,000 times per request has an N+1 problem.
-
Lock contention - many threads waiting on the same lock. The team that has a multi-threaded app with 1 hot lock has a serialization point.
-
GC pressure - JVM, Go, or Node.js process spending most of its time in garbage collection. The team that has a JVM with 99% CPU and frequent
Full GCevents has a heap sizing issue. -
High traffic - the process is doing the work it is supposed to do, just at a rate the system cannot handle. The fix: scale horizontally (more processes) or vertically (faster CPU).
The fix: kill, restart, optimize, or scale
Kill the offending process if it is not critical. The team that has a runaway test container takes it down.
Restart the process if it is critical and has a known bug. The team that has a long-running service with a memory or CPU leak schedules a daily restart until the bug is fixed.
Optimize the code. The team that has a slow database query adds an index, the team that has a slow algorithm refactors the hot path.
Scale horizontally. The team that has a web server at 90% CPU adds another instance behind a load balancer.
FAQ
What is a good load average?
Lower than the number of CPU cores. A 4-core box with load average 2 has headroom; load 4 is fully utilized; load 8 is oversubscribed. The team that monitors load averages sets alerts at cores * 0.8.
How do I find the process using the most CPU?
ps aux --sort=-%cpu | head -20 or top -o %CPU.
What is the difference between user CPU and system CPU?
User CPU is time spent in user-space code (your application). System CPU is time spent in kernel-space code (syscalls, interrupts, I/O). High system CPU usually means I/O or networking issues. The team that sees 50% system CPU has a kernel-level bottleneck.
How do I find a runaway process?
ps aux --sort=-%cpu | head. The team that sees a process at 100% CPU that should be idle has a runaway loop or a bug. Kill it, fix the bug, restart.
What is the difference between CPU usage and CPU load?
CPU usage is the percentage of one core being used. CPU load is the number of processes waiting for CPU time. A process at 100% usage contributes 1 to the load average. A 4-core box with 8 processes at 100% has load 8, even though all 4 cores are saturated.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: