Understanding performance¶
"The server is slow" always has a cause in one of four resources. The art is to find out which one is at its limit, instead of guessing.
The four resources¶
| Resource | Symptom | First look | In detail |
|---|---|---|---|
| CPU | High load, us or sy near 100 % |
top, vmstat |
sar -u, mpstat -P ALL |
| Memory | Swap is used, si/so not zero, OOM killer in the log |
free, vmstat |
sar -r, sar -W |
| Disk I/O | High wa (I/O wait), slow responses although CPU is idle |
vmstat column wa |
iostat -x, iotop |
| Network | Timeouts, retransmissions | ss -s |
sar -n DEV, sar -n TCP,ETCP |
Reading the load correctly¶
load average: 3.12, 1.85, 0.97
│ │ └─ average of the last 15 minutes
│ └─ last 5 minutes
└─ last minute
The load counts processes that run or wait, for CPU and for disk. Compare it with the
number of CPU cores (nproc): on tuxdex-srv with 2 cores, a load of 2 means fully
busy, 3 means processes are waiting. A high load with an idle CPU points to disk I/O.
The USE method¶
For every resource, check three things:
| Question | Example | |
|---|---|---|
| Utilization | How busy is it? | %util of a disk in iostat |
| Saturation | Is work waiting? | Run queue r in vmstat, aqu-sz in iostat |
| Errors | Are there errors? | dmesg, journalctl -k, smartctl |
Looking back in time¶
top and vmstat only show now. When the server was slow at 3 at night, only data
collected in the background helps. The package sysstat records CPU, memory, disk and
network every 10 minutes; sar reads it afterwards.
What you need to install¶
| Debian | Ubuntu | RHEL | Arch | |
|---|---|---|---|---|
| vmstat, sysctl | preinstalled (procps) |
preinstalled (procps) |
preinstalled (procps-ng) |
preinstalled (procps-ng) |
| iostat, sar, mpstat | sudo apt install sysstat |
sudo apt install sysstat |
sudo dnf install sysstat |
sudo pacman -S sysstat |
| Background collection for sar | /etc/default/sysstat: ENABLED="true" |
/etc/default/sysstat: ENABLED="true" |
sudo systemctl enable --now sysstat |
sudo systemctl enable --now sysstat |
| iotop | sudo apt install iotop-c |
sudo apt install iotop-c |
sudo dnf install iotop |
sudo pacman -S iotop-c |
| lsmod, modprobe | preinstalled (kmod) |
preinstalled (kmod) |
preinstalled (kmod) |
preinstalled (kmod) |
Measure before you tune
Change kernel parameters only when a measurement shows a bottleneck, and change one thing at a time. Most tuning guides from the internet were written for other kernels and other workloads.
Free memory isn't wasted
Linux uses unused memory as cache (buff/cache in free). A server with little
free but much available memory is fine. It only gets tight when available drops
and swap is used actively.