Reliability patterns keep a distributed call from turning a blip into an outage. They sit at process and network boundaries (HTTP, queues, DB). They are not GoF structural patterns and they do not make a wrong retry policy safe — they make a named policy.
Timeouts first
Every outbound call needs a deadline. Without one, a hung dependency holds threads, sockets, and user requests until the process dies. Set timeouts slightly below the caller’s own deadline so you fail before the user does. Distinguish connect timeout from overall request timeout.
Retry
- Retry only idempotent operations, or send an idempotency key the server honors.
- Backoff with jitter. Synchronized retries after a blip are a thundering herd.
- Cap attempts. Infinite retry is an outage with extra latency.
- Do not retry 400, 401, 403, 404, or 409. Retry 408 and 429 only with a timeout, jitter, and a cap, the same way you retry a 503.
Circuit breaker
After a streak of failures, stop calling for a cool-down (open), then try a probe (half-open). Protects your threads and gives the dependency room to recover. Pair with a fallback (stale cache, degraded feature, fail the request fast). A breaker that wraps a non-idempotent POST without care will hide duplicates or skip work you still owe the user.
Bulkhead
Isolate pools: separate thread/connection budgets per dependency so a slow payments API cannot starve catalog reads. In .NET this is often dedicated SocketsHttpHandler / HttpClient per outbound system plus limited concurrency (semaphore, queue length). In queues, separate consumers and prefetch.
Fallback and load shedding
When the breaker is open or the bulkhead is full, choose: cached value, reduced payload, or 503 with Retry-After. Load-shed before you are out of threads. Returning 200 with empty data can be worse than a clear failure if callers cache it as truth.
Example policy (shape, not a library pitch)
Set the client timeout to 2 seconds before the first send. Retry at most twice, and only on 408, 429, or 503. Open the breaker after 5 failures and allow one probe after 30 seconds. body returns a new HttpContent each attempt, because a content object can be sent once.
static async Task<HttpResponseMessage> ChargeAsync(
HttpClient client, Func<HttpContent> body, string idempotencyKey, CancellationToken ct)
{
for (var attempt = 1; attempt <= 3; attempt++)
{
using var request = new HttpRequestMessage(HttpMethod.Post, "/charges")
{
Content = body()
};
request.Headers.TryAddWithoutValidation("Idempotency-Key", idempotencyKey);
var response = await client.SendAsync(request, ct);
var code = (int)response.StatusCode;
if ((code != 408 && code != 429 && code != 503) || attempt == 3)
return response;
response.Dispose();
// backoff with jitter before the next attempt
}
throw new InvalidOperationException("unreachable");
}What breaks it
This loop posts a charge five times with no idempotency key and no look at the status. A 201 and a 400 are both tried again.
static async Task ChargeWithoutAKey(HttpClient client, object charge, CancellationToken ct)
{
for (var attempt = 0; attempt < 5; attempt++)
{
using var response = await client.PostAsync("/charges", JsonContent.Create(charge), ct);
}
}Pitfalls
- Retry + no timeout = slower outage.
- Retrying non-idempotent payments.
- One global HttpClient timeout for every dependency.
- Calling it “resilience” while the message handler still uses a shared DB connection pool of 2.
Related: large messages, performance.