Software metrics are numbers that change how you act. If a dashboard cannot change a decision this week, it is decoration. Monitor the user’s outcome first, then the resources that produce it.
Four golden signals (and cousins)
- Latency — p50 is vanity; p95/p99 and timeouts are what users feel. Split success vs error latency.
- Traffic — requests/s, queue depth, concurrent users. Know the shape of a normal Tuesday.
- Errors — rate and kind (5xx vs 429 vs client 4xx). Alert on burn rate, not a single blip.
- Saturation — CPU, thread pool, DB connections, disk. Saturation explains the other three.
RED (rate, errors, duration) and USE (utilization, saturation, errors) are the same idea from two ends.
SLI / SLO / SLA
An SLI is a measured indicator (e.g. “share of checkout HTTP 2xx under 300 ms”). An SLO is the target you hold yourselves to. An SLA is a contract with money attached. Do not write an SLA you cannot measure. Error budgets exist so you can ship: when the budget is burned, freeze features and fix reliability.
Application metrics worth having
Queue consumer lag, cache hit ratio, dependency error rate by name, GC pause, row counts you care about (orders/min). Cardinality kills: do not put user id on a Prometheus label.
Pitfalls
- CPU dashboard, no request latency.
- Averages hiding a bimodal distribution.
- Paging a human for a metric with no runbook.
- Vanity: lines of code, story points as “productivity.”
When the metric says “slow,” next: performance issues.