Brokers have size limits (SQS 256 KiB; Kafka often about 1 MiB via message.max.bytes; RabbitMQ 4 and later defaults to 16 MiB, and recent 3.x — about 3.8 through 3.13 — defaulted to 128 MiB). A message under the cap can still trip memory or disk alarms. A 30 MB JSON “event” is not an event; it is a file you shoved into a pipe. Treat the message as a pointer plus metadata once the body stops being small.
Claim check
Store the blob in object storage (S3, Azure Blob). Put the URI, content type, size, and checksum in the message. Consumers fetch the blob. Encrypt at rest; sign or expire URLs if the bucket is not private to the VPC.
{
"event": "InvoiceRendered",
"invoiceId": "…",
"blob": { "uri": "s3://…/inv.pdf", "sha256": "…", "bytes": 204812 }
}
Split and sequence
If you must stay on the broker, chunk with a correlation id and sequence numbers, and define what happens when chunk 3 of 10 never arrives. This is worse than claim check for most product code.
Why large messages hurt
- Head-of-line blocking: one fat message stalls a partition.
- GC and memory on consumers (especially if you deserialize into a giant object graph).
- Retries duplicate the blob transfer.
- Logs and tracing tools choke or leak PII.
What breaks it
Base64 makes the payload about a third larger and still pushes it through the broker. Use the claim-check message above: URI, content type, size, and checksum.
{
"event": "InvoiceRendered",
"pdfBase64": "JVBERi0xLj…"
}Pitfalls
- Base64-ing a file into JSON (33% larger, still hits the cap).
- Public S3 objects “because the worker needs them.”
- Mutating the blob after publish so checksums lie.
Pair with broker functions and idempotent consumers.