Understanding monitoring¶
Monitoring answers two questions: How are my servers doing right now? and Does someone get told when something breaks? Prometheus is the most common open source answer on Linux servers.
How Prometheus works¶
web01, db01, tuxdex-srv tuxdex-srv tuxdex-srv
┌──────────────────────┐ pull ┌──────────────────────┐ alerts ┌──────────────┐ mail
│ node_exporter :9100 │ ──────> │ Prometheus :9090 │ ───────> │ Alertmanager │ ─────> admin
│ /metrics │ │ stores time series, │ │ :9093 │
└──────────────────────┘ │ evaluates rules │ └──────────────┘
└──────────────────────┘
│
└──> Grafana (dashboards, optional)
| Part | Job |
|---|---|
| Exporter | Offers metrics as text on an HTTP page /metrics. The node exporter covers the server itself: CPU, RAM, disks, network |
| Prometheus | Fetches (scrapes) all targets every 15 seconds, stores the values and evaluates alert rules |
| Alertmanager | Receives firing alerts, groups them, silences them on request and sends mails or chat messages |
| Grafana | Draws dashboards from the data in Prometheus; not part of Prometheus itself |
Packages, services and ports¶
| Debian | Ubuntu | RHEL | Arch | |
|---|---|---|---|---|
| Prometheus | prometheus |
prometheus |
not in the repositories | prometheus |
| Node exporter | prometheus-node-exporter |
prometheus-node-exporter |
not in the repositories | prometheus-node-exporter |
| Alertmanager | prometheus-alertmanager |
prometheus-alertmanager |
not in the repositories | alertmanager |
| Started after install | yes | yes | see below | no: systemctl enable --now |
On RHEL, use the release archives from prometheus.io with your own systemd units, or the
container images prom/prometheus, prom/node-exporter and prom/alertmanager.
| Port | Service |
|---|---|
| 9090 | Prometheus (web interface and API) |
| 9093 | Alertmanager |
| 9100 | Node exporter |
A scrape configuration¶
/etc/prometheus/prometheus.yml (shortened):
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: node
static_configs:
- targets: ['localhost:9100', '192.0.2.21:9100', '192.0.2.22:9100']
After a change: promtool check config, then sudo systemctl reload prometheus.
PromQL in five queries¶
| Query | Result |
|---|---|
up |
1 for every reachable target, 0 for the others |
up == 0 |
Only the targets that are down |
node_load1 |
Load average of the last minute, per server |
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) |
RAM usage in percent |
rate(node_network_receive_bytes_total[5m]) |
Received bytes per second, averaged over 5 minutes |
You can type them into the web interface on port 9090 or run them with
promtool query instant.
An alert rule¶
/etc/prometheus/rules/node.yml:
groups:
- name: node
rules:
- alert: InstanceDown
expr: up == 0
for: 2m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} is down"
for: 2m means the condition must be true for two minutes before the alert fires. That
avoids mails for a short hiccup. The Alertmanager then decides who gets told; see
amtool.
Monitor the monitoring
If the Prometheus server itself fails, no alert arrives. A simple external check (a
second Prometheus, an uptime service or a cron job with curl
on /-/healthy) closes that gap.
Don't expose the ports
The metrics reveal a lot about your servers: versions, mount points, services. Allow ports 9100 and 9090 only from the Prometheus server and your admin network.