- Introduction — A Flood of Numbers
- Reading the Process Columns
- VIRT, RES, SHR — The Trap of the Three Memory Values
- The Process State Letters
- Load Average — It Is Not CPU Usage
- The Meter Colors
- What Zombie Processes Really Are
- Copy-on-Write and the RES Illusion
- Practical Diagnostic Recipes
- Conclusion
- References
Introduction — A Flood of Numbers
Open htop or top on a Linux server for the first time and you are overwhelmed by a flood of numbers and colors. Most people glance at the CPU and memory percentages and ignore the rest, but each of those numbers holds information you need to diagnose the state of the system. Prompted by Pēteris Ņikiforovs's beloved explainer "Explanation of everything you can see in htop" recently resurfacing near the top of Hacker News, this article takes those fields apart one by one.
Reading the Process Columns
Let us start with what each column in the process list means.
- PID: the process ID, a unique number assigned when a process starts.
- USER: the owner of the process — the user who launched it.
- PRI and NI: priority and nice value. The relationship is
PR = 20 + NI. Nice ranges from -20 (highest priority) to 19 (lowest priority); the lower the nice, the more CPU the process gets. As a rough heuristic, each nice step shifts about 10% of CPU time. - VIRT / RES / SHR: three memory values, covered separately below.
- S: the process state, also covered below.
- %CPU: CPU usage.
- %MEM: memory usage, computed as
RES / total RAM. - TIME+: the cumulative CPU time the process has used so far.
VIRT, RES, SHR — The Trap of the Three Memory Values
The memory columns are the most commonly misunderstood part. The three values mean completely different things.
- VIRT (virtual memory): the entire virtual address space the process has reserved. This includes regions it never actually uses, shared libraries, memory-mapped files, and pages swapped out to disk. So VIRT is usually inflated and nearly useless as an indicator of real memory usage.
- RES (resident memory): the non-swapped physical RAM the process is actually using. If you want to know "how much RAM is this process eating right now," this is the value to look at.
- SHR (shared memory): memory that could be shared with other processes — shared libraries are the classic example.
An important practical lesson follows: when hunting for a memory-hungry process, sort by RES, not VIRT. Sort by VIRT and a process that reserved a large address space but barely uses memory floats to the top, leading you astray.
The Process State Letters
The S column shows a one-letter process state. The letters mean:
R— running or runnable. Executing on a CPU or waiting its turn.S— interruptible sleep. Sleeping while waiting for an event, but wakeable by a signal. Most processes are in this state normally.D— uninterruptible sleep. Usually blocked on disk I/O, and cannot be woken even by a signal.Z— zombie. Terminated but not yet reaped by its parent.T— stopped by a job-control signal (e.g. Ctrl+Z).t— stopped by a debugger/tracer.
Load Average — It Is Not CPU Usage
The most widely misunderstood metric is the load average. People often think it equals "CPU usage," but it does not.
First, load average is not a simple average but an exponentially damped moving average. Three values are shown — over 1, 5, and 15 minutes — weighting recent values more heavily. The 1-minute figure is roughly 63% of the last minute's load plus 37% of what came before.
Second, and more important — load counts not only running (R) processes but also processes in uninterruptible sleep (the D state). Because of this, a high load does not necessarily mean the CPU is busy. Load spikes even when many processes are waiting on disk or network I/O.
Third, load must be interpreted against the core count. On a machine with N cores, a load of N means roughly full utilization. Load 4 on a 4-core box is saturation; load 4 on an 8-core box is only half utilized.
Combining these gives a powerful diagnostic rule: if load is high but %CPU is low, it is almost certainly I/O wait. The place to start is looking for D-state processes.
The Meter Colors
The bar meters at the top of htop convey information through color too.
- CPU meter: blue for low-priority (positively niced) threads, green for normal-priority threads, red for kernel threads. (htop also breaks out categories like virtualization and I/O wait in finer detail.)
- Memory meter: green for used memory, blue for buffers, orange for cache.
The key is not to mistake buffers and cache for "used memory." Buffers and cache are regions the kernel borrows temporarily for performance, and they are returned immediately when an application needs memory. So the value to watch is not "used" but available memory. This is why a Linux screen showing "memory almost full" is usually not a problem — memory filled by cache is, in effect, free memory that can be reclaimed on demand.
What Zombie Processes Really Are
Zombie (Z) processes are often misunderstood. A few facts:
A zombie is an already-dead process. It has finished executing, but its parent has not yet reaped its exit status, so only an entry remains in the process table. Because of this, a zombie consumes no memory at all — it occupies only a single slot in the process table.
Also, a zombie cannot be killed. To terminate a process you send it a signal, and only a live process can receive a signal. Sending kill -9 to a zombie that is already dead does nothing. The only way a zombie disappears is for its parent to reap it. If the parent dies first, the zombie is adopted by init (PID 1), which reaps it instead.
So there is no need to panic over a few zombies. If zombies keep piling up, that points less to the zombies themselves being a problem than to a bug in a parent process that fails to reap its children.
Copy-on-Write and the RES Illusion
One subtle phenomenon to add. When a process creates a child with fork, Linux uses copy-on-write. Parent and child share the same physical memory pages, and a page is copied only when one of them writes to it.
Because of this, htop can show both parent and child with the full RES each. It looks like double the memory, but the physical memory is shared, so the total is not that large. Keep this illusion in mind when looking at the memory of a program that forks heavily.
Practical Diagnostic Recipes
Applying all this to real troubleshooting:
- "Load is high but the CPU is idle" → I/O wait. Look for
D-state processes and suspect a disk or network bottleneck. - Finding the memory culprit → sort by RES, not VIRT.
- Finding a runaway thread → turn on thread display in htop and watch for an item with persistently high
%CPU. - "Memory is full" → look at available, not used. Memory filled by cache is not a problem.
If you want to get hands-on with these tools in the browser, open this site's Linux terminal or web terminal, and to explore how kernel parameters bear on this behavior, the kernel parameter explorer.
Conclusion
The numbers in htop and top look like a cipher at first, but once you know what each field means they become a powerful diagnostic tool for reading the state of a system. In particular, understanding just three things — the difference between VIRT and RES, that load includes I/O wait, and that cache is effectively free memory — already solves half of server troubleshooting.