Memory usage high on Linux is debugged in three steps: confirm the issue with free -h, find the top consumers with ps aux --sort=-%mem or top -o %MEM, then drill into the worst offender with pmap for a per-process memory map. The team that finds the leaking process fixes the leak. The team that just reboots has a recurring problem - the leak is still there, just hidden until the next OOM.
Table of contents
- Step 1: confirm the issue
- Step 2: find the top consumers
- Step 3: drill into the process
- Common causes
- The fix: kill, restart, configure, or add memory
- The OOM killer
- How this fits the rest of the stack
- FAQ
Step 1: confirm the issue
free -h
Output: total, used, free, shared, buff/cache, available. The available column is what matters - it accounts for reclaimable cache and shows what the system can actually allocate to new processes.
If available is low (under 10% of total) and used is high (over 90% of total), the system is under memory pressure. The team that sees swap activity in vmstat 1 is definitely under pressure.
vmstat 1 10
Look at si and so (swap in / swap out). Non-zero values mean the system is swapping - slow disk I/O is the user-visible symptom.
Step 2: find the top consumers
ps aux --sort=-%mem | head -20
Output: the top 20 processes by memory usage. The team that sees a process at 40% of total memory has found the offender.
Or interactive: top -o %MEM (sorts by memory). Press M to sort by memory in top itself.
Or for a tree view (parent + children): ps auxf --sort=-%mem | head -30. The team that has a parent process spawning many children sees the total impact in the tree.
Step 3: drill into the process
For a per-process memory map:
sudo pmap -x <pid> | sort -k3 -n -r | head -20
Output: the largest memory regions in the process. The team that sees a 4 GB heap region has a process that allocated 4 GB of memory - the question is why.
For per-thread memory in a process:
cat /proc/<pid>/status
Look at VmRSS (resident set size - the actual memory used), VmSize (virtual size), VmPeak (peak virtual size). The team that compares VmRSS to VmSize sees the difference between allocated and actually-used memory.
Common causes
-
Memory leak - the process allocates memory and never frees it. Restarting the process frees the memory; the leak returns over time. The team that has a process that grows by 100 MB/hour has a memory leak.
-
Heap too large - the process is configured to use a lot of memory (e.g., JVM
-Xmx, Node.js--max-old-space-size). The fix: reduce the heap size, or give the process more memory. -
Cached pages - the OS is using free memory for disk cache. This is normal and good; the team that sees
buff/cachehigh butavailablenormal is fine. -
Many small processes - a process that has spawned many children (e.g., a web server with 1000 workers). The total memory is large; the per-process memory is small. The team that has 1000 processes at 50 MB each has 50 GB used.
The fix: kill, restart, configure, or add memory
Kill the offending process if it is not critical. The team that has a runaway test container takes it down.
Restart the process if it is critical and has a known leak. The team that has a long-running service with a leak schedules a daily restart until the leak is fixed.
Configure the process to use less memory. The team that has a JVM at -Xmx16g on a 16 GB machine has no headroom - reduce to -Xmx12g.
Add memory to the host. The team that has a 4 GB VPS and a 6 GB workload needs a bigger VPS.
The OOM killer
When the system runs out of memory, the OOM (Out-Of-Memory) killer kicks in and kills a process. Check the kernel log:
dmesg | grep -i 'killed process'
The team that has a process that mysteriously dies at 3 AM checks dmesg for OOM events. The killed process is the one with the highest oom_score at the moment of OOM - usually the largest memory consumer.
To prevent future OOMs: add memory, configure the process to use less, or use cgroups to limit memory per process.
FAQ
Why is my Linux server using so much memory?
Because the OS uses free memory for disk cache. The buff/cache column in free -h shows this. The available column is what matters - it accounts for reclaimable cache.
How do I find the process using the most memory?
ps aux --sort=-%mem | head -20 or top -o %MEM.
What is the difference between RSS and VSZ?
VSZ (virtual size) is the total memory the process has access to, including swapped-out and unused. RSS (resident set size) is the actual physical memory the process is using. The team that cares about physical memory uses RSS, not VSZ.
How do I find a memory leak?
Watch the process’s RSS over time (ps -p <pid> -o rss= --no-headers in a loop, or top -p <pid>). If the RSS grows steadily and never drops, you have a leak. The team that has a Node.js process that grows from 200 MB to 4 GB over a week has a leak.
What is the OOM killer?
When the system runs out of memory, the kernel kills the process with the highest oom_score. The team that has a process that dies randomly checks dmesg | grep -i 'killed process' to see if OOM was the cause.
How do I prevent the OOM killer from killing my process?
Set oom_score_adj to -1000: echo -1000 > /proc/<pid>/oom_score_adj. The process becomes the last to be killed. The team that has a critical database uses this.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: