{"id":12,"date":"2024-08-24T08:56:15","date_gmt":"2024-08-24T08:56:15","guid":{"rendered":"https:\/\/contentlabstudy.com\/soft\/?p=12"},"modified":"2026-09-24T21:15:22","modified_gmt":"2026-09-24T20:15:22","slug":"reliability-design-patterns","status":"publish","type":"post","link":"https:\/\/contentlabstudy.com\/soft\/reliability-design-patterns\/","title":{"rendered":"Reliability Design Patterns"},"content":{"rendered":"<p><strong>Reliability patterns<\/strong> keep a distributed call from turning a blip into an outage. They sit at process and network boundaries (HTTP, queues, DB). They are not GoF structural patterns and they do not make a wrong retry policy safe \u2014 they make a <em>named<\/em> policy.<\/p>\n<h2>Timeouts first<\/h2>\n<p>Every outbound call needs a deadline. Without one, a hung dependency holds threads, sockets, and user requests until the process dies. Set timeouts slightly below the caller\u2019s own deadline so you fail before the user does. Distinguish connect timeout from overall request timeout.<\/p>\n<h2>Retry<\/h2>\n<ul>\n<li>Retry only <strong>idempotent<\/strong> operations, or send an idempotency key the server honors.<\/li>\n<li>Backoff with jitter. Synchronized retries after a blip are a thundering herd.<\/li>\n<li>Cap attempts. Infinite retry is an outage with extra latency.<\/li>\n<li>Do not retry 400, 401, 403, 404, or 409. Retry 408 and 429 only with a timeout, jitter, and a cap, the same way you retry a 503.<\/li>\n<\/ul>\n<h2>Circuit breaker<\/h2>\n<p>After a streak of failures, <strong>stop calling<\/strong> for a cool-down (open), then try a probe (half-open). Protects <em>your<\/em> threads and gives the dependency room to recover. Pair with a fallback (stale cache, degraded feature, fail the request fast). A breaker that wraps a non-idempotent POST without care will hide duplicates or skip work you still owe the user.<\/p>\n<h2>Bulkhead<\/h2>\n<p>Isolate pools: separate thread\/connection budgets per dependency so a slow payments API cannot starve catalog reads. In .NET this is often dedicated <code>SocketsHttpHandler<\/code> \/ <code>HttpClient<\/code> per outbound system plus limited concurrency (semaphore, queue length). In queues, separate consumers and prefetch.<\/p>\n<h2>Fallback and load shedding<\/h2>\n<p>When the breaker is open or the bulkhead is full, choose: cached value, reduced payload, or 503 with Retry-After. Load-shed <em>before<\/em> you are out of threads. Returning 200 with empty data can be worse than a clear failure if callers cache it as truth.<\/p>\n<h2>Example policy (shape, not a library pitch)<\/h2>\n<p>Set the client timeout to 2 seconds before the first send. Retry at most twice, and only on 408, 429, or 503. Open the breaker after 5 failures and allow one probe after 30 seconds. <code>body<\/code> returns a new <code>HttpContent<\/code> each attempt, because a content object can be sent once.<\/p>\n<pre><code>static async Task&lt;HttpResponseMessage&gt; ChargeAsync(\n    HttpClient client, Func&lt;HttpContent&gt; body, string idempotencyKey, CancellationToken ct)\n{\n    for (var attempt = 1; attempt &lt;= 3; attempt++)\n    {\n        using var request = new HttpRequestMessage(HttpMethod.Post, \"\/charges\")\n        {\n            Content = body()\n        };\n        request.Headers.TryAddWithoutValidation(\"Idempotency-Key\", idempotencyKey);\n        var response = await client.SendAsync(request, ct);\n        var code = (int)response.StatusCode;\n        if ((code != 408 &amp;&amp; code != 429 &amp;&amp; code != 503) || attempt == 3)\n            return response;\n        response.Dispose();\n        \/\/ backoff with jitter before the next attempt\n    }\n\n    throw new InvalidOperationException(\"unreachable\");\n}<\/code><\/pre>\n<h2>What breaks it<\/h2>\n<p>This loop posts a charge five times with no idempotency key and no look at the status. A 201 and a 400 are both tried again.<\/p>\n<pre><code>static async Task ChargeWithoutAKey(HttpClient client, object charge, CancellationToken ct)\n{\n    for (var attempt = 0; attempt &lt; 5; attempt++)\n    {\n        using var response = await client.PostAsync(\"\/charges\", JsonContent.Create(charge), ct);\n    }\n}<\/code><\/pre>\n<h2>Pitfalls<\/h2>\n<ul>\n<li>Retry + no timeout = slower outage.<\/li>\n<li>Retrying non-idempotent payments.<\/li>\n<li>One global HttpClient timeout for every dependency.<\/li>\n<li>Calling it \u201cresilience\u201d while the message handler still uses a shared DB connection pool of 2.<\/li>\n<\/ul>\n<p>Related: <a href=\"\/soft\/message-brokers-and-large-messages\/\">large messages<\/a>, <a href=\"\/soft\/software-performance-issues\/\">performance<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Reliability patterns keep a distributed call from turning a blip into an outage. They sit at process and network boundaries [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-12","post","type-post","status-publish","format-standard","hentry","category-software-design"],"_links":{"self":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":8,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/12\/revisions"}],"predecessor-version":[{"id":3222,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/12\/revisions\/3222"}],"wp:attachment":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/media?parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/categories?post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/tags?post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}