A performance issue is a missed latency, throughput, or cost budget — not “the server feels slow.” Start from a number (p95 checkout, jobs/hour, $ of DB CPU). Then measure, change one thing, measure again.
Find the dominant cost
- Wait vs work: is the thread on-CPU or blocked on I/O? Profilers and
EXPLAIN ANALYZEdisagree with guesses. - N+1 and chatty I/O: 80 HTTP calls in a page render. Batch, join, or cache with a defined TTL and invalidation.
- Locking: serializing on one row or one global mutex. See SQL.
- Payload size: 5 MB JSON for a list view. Paginate; don’t serialize the universe.
- GC / allocations: hot paths that allocate per request in a managed runtime.
Cache is a correctness problem
Cache only with a story for stale reads. Key design, TTL, and stampede (dogpile) matter. Caching a wrong authorization decision is a security bug. See reliability for timeouts so a slow cache is not an infinite wait.
Capacity vs algorithm
Sometimes you buy a larger instance. Sometimes you stop doing O(n²) on a 100k list. Do not scale out a quadratic algorithm and call it architecture.
Example
EXPLAIN ANALYZE shows a sequential scan of hundreds of milliseconds. The predicate cannot use a plain index on email.
-- EXPLAIN ANALYZE: Seq Scan
SELECT id FROM customer WHERE lower(email) = lower($1);What breaks it
The team rewrites a LINQ micro-benchmark while that scan is the request. Store the normalized email, index the column the predicate uses, measure once, then change one thing.
SELECT id FROM customer WHERE email = $1;Pitfalls
- Optimizing before a profile (especially micro-optimizing C# that is waiting on SQL).
- Load tests that never look like production traffic (no think time, no cache, one user id).
- Indexes on every column “just in case” — write amplification, unused indexes.