Software metrics are numbers that change how you act. If a dashboard cannot change a decision this week, it is decoration. Monitor the user’s outcome first, then the resources that produce it.
Four golden signals (and cousins)
- Latency — p50 is the typical request; pages and SLOs use p95/p99 and timeouts. Split success vs error latency.
- Traffic — requests/s, queue depth, concurrent users. Know the shape of a normal Tuesday.
- Errors — rate and kind (5xx vs 429 vs client 4xx). Alert on burn rate, not a single blip.
- Saturation — CPU, thread pool, DB connections, disk. Saturation explains the other three.
RED (rate, errors, duration) and USE (utilization, saturation, errors) are the same idea from two ends.
SLI / SLO / SLA
An SLI is a measured indicator (e.g. “share of checkout HTTP 2xx under 300 ms”). An SLO is the target you hold yourselves to. An SLA is a contract with money attached. Do not write an SLA you cannot measure. Error budgets exist so you can ship: when the budget is burned, freeze features and fix reliability.
Application metrics worth having
Queue consumer lag, cache hit ratio, dependency error rate by name, GC pause, row counts you care about (orders/min). Cardinality kills: do not put user id on a Prometheus label.
Example
An SLI you can page on: the share of POST /checkout responses that are 2xx and finish in under 300 ms.
What breaks it
A user_id label on the request metric creates one series per user. Keep route and status.
http_request_duration_seconds{user_id="1842"}
http_request_duration_seconds{route="POST /checkout", status="200"}Pitfalls
- CPU dashboard, no request latency.
- Averages hiding a bimodal distribution.
- Paging a human for a metric with no runbook.
- Vanity: lines of code, story points as “productivity.”
When the metric says “slow,” next: performance issues.