System monitoring tools split into three categories: interactive triage tools (top, htop, atop) for live debugging, batch sampling tools (vmstat, iostat, mpstat, sar) for scripted capture, and full observability stacks (Prometheus + node_exporter + Grafana) for trend analysis and alerting. The team that uses htop for ‘what is happening right now’, vmstat for ‘what happened over the last 5 minutes’, and node_exporter for ‘what is the trend over the last 30 days’ has the right tool for each job. The team that only uses top has live data but no history; the team that only uses Prometheus has trends but no live triage.
Table of contents
- Interactive triage tools
- Batch sampling tools
- Full observability stacks
- Specialty tools
- Choosing the right tool
- FAQ
Interactive triage tools
top is the universal interactive monitor, available on every Unix system since the 1980s. Shows process list sorted by CPU usage, with memory, load average, and uptime. The team that SSHes into a server and runs top sees the live process state. Shortcuts: P to sort by CPU, M to sort by memory, k to kill a process, q to quit.
htop is top with better visuals and easier interaction. Color-coded CPU bars per core, mouse-clickable process list, easier process kill (F9), tree view (F5), and search (F3). The team that runs htop instead of top gets a better triage experience. Install via apt install htop on Ubuntu.
atop is the advanced interactive monitor for post-mortem analysis. Logs resource usage to a file (when run as a daemon) so the team can rewind to see what happened at 2am. The team that runs atop as a daemon (systemctl enable atop) has historical data for incident investigation. Install via apt install atop.
Batch sampling tools
vmstat reports virtual memory statistics. Run with vmstat 1 for 1-second sampling. Output columns: procs (r=runnable, b=blocked), memory (swpd, free, buff, cache), swap (si, so), io (bi, bo), system (in, cs), cpu (us, sy, id, wa, st). The team that watches r (run queue) and wa (iowait) catches CPU saturation and disk pressure.
iostat reports disk I/O statistics. Run with iostat -x 1 for extended stats. Output includes await (average I/O wait time in ms), svctm (service time), %util (disk utilization). The team that watches await and %util catches disk bottlenecks. Install via apt install sysstat.
mpstat reports per-CPU statistics. Run with mpstat -P ALL 1 for all CPUs. Output includes %usr, %sys, %iowait, %idle, %steal per CPU. The team that sees steal time high on some CPUs but not others has uneven CPU contention.
sar is the System Activity Reporter - logs system activity over time. Run as a daemon (sysstat package) to log every 10 minutes. The team that runs sar has historical data for trend analysis and capacity planning.
Full observability stacks
Prometheus + node_exporter + Grafana is the open-source de facto stack. node_exporter runs on each host, exposing metrics in Prometheus format. Prometheus scrapes node_exporter every 15 seconds. Grafana queries Prometheus for dashboards. The team that runs this stack has metrics for the whole fleet with a unified query language (PromQL).
Datadog, New Relic, Honeycomb are managed alternatives. Per-host cost (~$15-30/host/month) but zero setup. The team that uses managed monitoring has faster onboarding; the team that uses self-hosted has lower per-host cost at scale.
Netdata is a single-binary monitoring agent with a web UI. Real-time per-second metrics, hundreds of charts out of the box, free for self-hosted. The team that runs Netdata has immediate visibility without setting up Prometheus + Grafana.
Specialty tools
iotop shows disk I/O per process (like top for disk). Useful when a specific process is hammering the disk. The team that sees high iostat %util but cannot identify the process runs iotop to find it.
iftop shows network I/O per connection (like top for network). Useful when a specific connection is consuming bandwidth. The team that sees high network usage but cannot identify the source runs iftop.
nethogs shows network I/O per process. Like iftop but per-process instead of per-connection. The team that sees high network usage and wants to know which process runs nethogs.
strace traces system calls of a process. Shows every file open, every network connect, every signal. The team that debugs ‘why is this process hanging’ runs strace to see what it’s waiting on.
lsof lists open files. The team that gets ‘port already in use’ runs lsof -i :8000 to find which process holds port 8000.
Choosing the right tool
Live triage on a single server: htop + iotop + iftop. The team that SSHes into a server to debug a current issue starts with htop (overall state), drops to iotop (disk issue) or iftop (network issue) as needed.
Sampling over time on a single server: vmstat + iostat + mpstat piped to a file. The team that captures 5 minutes of system state runs vmstat 1 300 > vmstat.log and reviews the file later.
Trend analysis and alerting across the fleet: Prometheus + node_exporter + Grafana. The team that monitors 10+ hosts uses a centralized stack. The team that monitors 2-3 hosts uses local tools + cron-emailed reports.
FAQ
What is the difference between top and htop?
Both show live process state. htop has color-coded CPU bars per core, mouse support, easier process management, and tree view. top is available everywhere; htop needs installation (apt install htop). The team that uses htop has a better experience; the team stuck with top (no install permission) gets the job done with shortcuts.
How do I capture system stats over time?
Run vmstat 1 > vmstat.log or iostat -x 1 > iostat.log in the background. The team that captures stats during an incident has data for post-mortem analysis. For long-term, use sar (sysstat package) or Prometheus node_exporter.
What is iowait?
iowait is the percentage of CPU time spent waiting for disk I/O. High iowait (above 20% sustained) means the disk is the bottleneck. The team that sees high iowait checks disk type (HDD vs SSD), disk utilization (%util in iostat), and the disk queue depth.
What is steal time?
Steal time is the percentage of CPU time the VM wanted but the hypervisor gave to another VM. High steal time (above 5%) on a VM means the host is oversubscribed. The team that monitors steal time catches oversubscription before users complain about slow VMs.
How do I find which process is using a port?
lsof -i :8000 on Linux/macOS, netstat -ano | findstr :8000 on Windows. The team that gets ‘port already in use’ uses these to find and kill the offending process.
What is the best free monitoring tool?
For self-hosted: Prometheus + Grafana + node_exporter is the standard. For a single-binary solution: Netdata. The team that picks based on scale: Netdata for 1-5 hosts, Prometheus stack for 5+, commercial (Datadog) for 50+.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: