An OOM error in the cloud is usually the container, the host, the kernel, the application, the memory leak, or the workload. The OOM killer is the kernel’s last resort when the system is out of memory. The fix is to find what is using the memory and address the cause, not to resize the host.
Table of contents
- Find the killer
- Check the container
- Check the host
- Check the kernel
- Check the application
- Check the workload
- FAQ
Find the killer
The first step is to confirm the OOM killer is the cause. On Linux:
dmesg | grep -i oomshows the recent OOM kills./var/log/messagesor/var/log/syslogshows the system log, which includes the OOM killer events.
On Kubernetes, the kubelet logs the OOM kill event to the event log. kubectl describe pod <pod> shows the recent events, including the OOM kill.
Check the container
The first thing to check: is the container’s memory limit set? On Kubernetes, the container’s memory limit is the most common cause of an OOM kill. The container hits the limit, the kernel OOM-kills the process, the kubelet restarts the container.
The fix: either raise the memory limit (if the workload genuinely needs more memory) or fix the workload (if the workload has a memory leak).
Check the host
The second thing to check: is the host’s memory exhausted? The container limit is not the only limit; the host’s memory is also a limit. The team that has 10 containers each with 8 GB of memory limit on a 16 GB host is going to OOM-kill the workloads.
The fix: either reduce the container’s memory limit, reduce the number of containers on the host, or increase the host’s memory.
Check the kernel
The third thing to check: is the kernel’s overcommit behavior causing the OOM? The Linux kernel has a vm.overcommit_memory setting that controls how aggressively the kernel allocates memory. The default is 0 (heuristic overcommit); the team that has set it to 1 (always overcommit) is going to OOM-kill the workloads when the system actually runs out of memory.
The fix: either reduce the workload’s memory usage or change the overcommit setting to 0 (heuristic) or 2 (never overcommit).
Check the application
The fourth thing to check: does the application have a memory leak? A memory leak is when the application allocates memory and does not free it. The pattern: the application starts at 100 MB, grows to 1 GB over a few hours, the container hits the memory limit, the OOM killer fires.
The fix: profile the application, find the leak, and fix it. The most common cause of memory leaks in modern applications: an unbounded cache, an unbounded queue, an unbounded list of pending requests.
Check the workload
The fifth thing to check: has the workload grown beyond the memory limit? The workload that is sized for 1000 requests per minute and is now at 10000 requests per minute is going to use 10x the memory. The OOM kill is the symptom; the growth is the cause.
The fix: either raise the memory limit (if the growth is expected) or reduce the workload (if the growth is unexpected). The team that raises the memory limit without understanding the cause is going to be back here in a month.
FAQ
Why is my cloud workload OOM-killed?
Usually the container’s memory limit, the host’s memory exhaustion, the kernel’s overcommit, the application’s memory leak, or the workload’s growth. Check those before resizing.
How do I confirm the OOM killer is the cause?
On Linux: dmesg | grep -i oom. On Kubernetes: kubectl describe pod <pod>. The OOM killer logs the event to the system log and the kubelet logs the event to the pod’s events.
How do I fix a Kubernetes OOM kill?
Either raise the memory limit (if the workload genuinely needs more memory) or fix the workload (if it has a memory leak). The most common cause is a memory leak in the application.
What is the right Kubernetes memory limit?
The right answer depends on the workload. Start with 1-2x the workload’s average memory usage and adjust based on the OOM kill rate. The team that sets the limit too low gets OOM kills; the team that sets it too high wastes memory.
How do I find a memory leak in my application?
Profile the application with a memory profiler. For Java: VisualVM or JFR. For Python: tracemalloc or memory_profiler. For Go: pprof. The profile shows the allocation that is not being freed.
If you are sizing the memory tier for a new project, the RunxBuild hosting calculator is the place to model the line items. The memory, the containers, the limits, the workload - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers. The RunxBuild dashboard is where the team sees the actual memory usage in one place.