{"id":2920,"date":"2024-08-24T12:13:28","date_gmt":"2024-08-24T11:13:28","guid":{"rendered":"https:\/\/contentlabstudy.com\/soft\/?p=2920"},"modified":"2026-09-24T21:15:37","modified_gmt":"2026-09-24T20:15:37","slug":"message-brokers-and-large-messages","status":"publish","type":"post","link":"https:\/\/contentlabstudy.com\/soft\/message-brokers-and-large-messages\/","title":{"rendered":"Message Brokers and Large Messages"},"content":{"rendered":"<p>Brokers have <strong>size limits<\/strong> (SQS 256 KiB; Kafka often about 1 MiB via <code>message.max.bytes<\/code>; RabbitMQ 4 and later defaults to 16 MiB, and recent 3.x \u2014 about 3.8 through 3.13 \u2014 defaulted to 128 MiB). A message under the cap can still trip memory or disk alarms. A 30 MB JSON \u201cevent\u201d is not an event; it is a file you shoved into a pipe. Treat the message as a <strong>pointer plus metadata<\/strong> once the body stops being small.<\/p>\n<h2>Claim check<\/h2>\n<p>Store the blob in object storage (S3, Azure Blob). Put the URI, content type, size, and checksum in the message. Consumers fetch the blob. Encrypt at rest; sign or expire URLs if the bucket is not private to the VPC.<\/p>\n<pre><code>{\n  \"event\": \"InvoiceRendered\",\n  \"invoiceId\": \"\u2026\",\n  \"blob\": { \"uri\": \"s3:\/\/\u2026\/inv.pdf\", \"sha256\": \"\u2026\", \"bytes\": 204812 }\n}\n<\/code><\/pre>\n<h2>Split and sequence<\/h2>\n<p>If you must stay on the broker, chunk with a correlation id and sequence numbers, and define what happens when chunk 3 of 10 never arrives. This is worse than claim check for most product code.<\/p>\n<h2>Why large messages hurt<\/h2>\n<ul>\n<li>Head-of-line blocking: one fat message stalls a partition.<\/li>\n<li>GC and memory on consumers (especially if you deserialize into a giant object graph).<\/li>\n<li>Retries duplicate the blob transfer.<\/li>\n<li>Logs and tracing tools choke or leak PII.<\/li>\n<\/ul>\n<h2>What breaks it<\/h2>\n<p>Base64 makes the payload about a third larger and still pushes it through the broker. Use the claim-check message above: URI, content type, size, and checksum.<\/p>\n<pre><code>{\n  \"event\": \"InvoiceRendered\",\n  \"pdfBase64\": \"JVBERi0xLj\u2026\"\n}<\/code><\/pre>\n<h2>Pitfalls<\/h2>\n<ul>\n<li>Base64-ing a file into JSON (33% larger, still hits the cap).<\/li>\n<li>Public S3 objects \u201cbecause the worker needs them.\u201d<\/li>\n<li>Mutating the blob after publish so checksums lie.<\/li>\n<\/ul>\n<p>Pair with <a href=\"\/soft\/message-broker-functions\/\">broker functions<\/a> and idempotent consumers.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Brokers have size limits (SQS 256 KiB; Kafka often about 1 MiB via message.max.bytes; RabbitMQ 4 and later defaults to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[3],"tags":[],"class_list":["post-2920","post","type-post","status-publish","format-standard","hentry","category-software-development"],"_links":{"self":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/2920","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/comments?post=2920"}],"version-history":[{"count":4,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/2920\/revisions"}],"predecessor-version":[{"id":3228,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/posts\/2920\/revisions\/3228"}],"wp:attachment":[{"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/media?parent=2920"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/categories?post=2920"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/contentlabstudy.com\/soft\/wp-json\/wp\/v2\/tags?post=2920"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}