A client asked us recently whether they needed Kafka. They had one producer, one consumer, and roughly four thousand messages a day. The honest answer was no, and the conversation that followed was more useful than the technology itself. Choosing async infrastructure is one of those decisions that is cheap to get right early and expensive to unwind once three services depend on it.
This is aimed at the point where a team notices a request handler is doing too much synchronously: sending an email, calling a third-party API, resizing an image, updating a search index. The instinct is to reach for a queue. Before doing that, it is worth working out which problem is actually being solved, because “queue” covers several quite different tools with different failure modes.
Start by asking whether you need a broker at all
If the system already has a relational database, and the volume is modest, a table can do the job. A row with a status column (pending, processing, done, failed), a worker that polls with SELECT ... FOR UPDATE SKIP LOCKED, and a cron job to retry failures will get a small product a long way. It is boring, it is easy to inspect with SQL when something goes wrong, and it does not add an operational dependency.
The database-as-queue approach breaks down once you need consumer groups (several independent services reacting to the same event), very high throughput, or ordering guarantees across partitions. That is the point to look at a dedicated broker, not before.
The three shapes of the problem
Most requests we see fall into one of three categories, and each points at a different tool.
Background jobs
Send this email, generate this PDF, call this webhook with retries. One producer, one type of consumer, no need to replay history. This is what Amazon SQS, or an equivalent managed queue on another cloud, is for. It is fully managed, it has no cluster to patch, and the mental model is small: messages go in, a worker pulls them out, you delete them when done. The cost of not needing to think about it is difficult to overstate for a small team.
Routing and delivery control
Work that needs priority tiers, delayed delivery, per-message TTLs, or dead-lettering with custom requeue rules, and where different message types need to fan out to different queues based on content. This is RabbitMQ’s territory. It costs more in operational attention than SQS, whether self-hosted or run through a managed provider, but the routing flexibility (exchanges, bindings, per-queue policies) is hard to replicate with a plain queue.
Event streams multiple services need to read independently
Order placed, user signed up, sensor reading recorded: an event that several consumers need to react to now, and that you may want to reprocess later (rebuild a search index, backfill a new analytics pipeline, debug what happened). This is what Kafka, or a managed equivalent such as Confluent Cloud or Amazon MSK, is built for. The retention and replay model is the actual feature; if nothing in your system will ever need to reread history, you are paying for a capability you are not using.
What running it actually costs
The comparison that matters for a small team is not throughput benchmarks, it is who is on call. A self-hosted Kafka cluster needs someone who understands partitions, replication factor, consumer lag, and broker disk pressure, and who will be paged when a partition leader election goes wrong at 2am. If nobody on the team has run Kafka in production before, budget for either a managed offering or for the learning curve to happen on a system that matters, which is a worse place to learn it.
RabbitMQ is lighter to operate but still needs monitoring for queue depth, memory high-water marks, and disk alarms, particularly if messages persist. SQS removes almost all of this: the operational surface is IAM permissions and a visibility timeout setting.
None of this means avoid the more capable tool. It means the decision should include an honest answer to “who fixes it when it degrades, and do they already know how.”
Ordering, delivery guarantees, and the questions worth asking upfront
Before picking a tool, it is worth writing down answers to a short list of questions, because the answers constrain the choice more than any feature comparison:
Does message order matter within a given key (a user’s events, an order’s state transitions), or is order irrelevant?
Is at-least-once delivery acceptable, meaning consumers must be idempotent, or does the business logic assume exactly-once?
Will more than one service ever need to consume the same message independently?
Does anything downstream need to replay history, or is “process once and discard” fine?
What is the actual message volume, not the volume you might have in two years?
“Exactly-once” is worth a specific warning: almost no broker guarantees it end to end without cooperation from the consumer. Design consumers to be idempotent (an operation that can safely run twice) regardless of which broker you pick, because at-least-once delivery is the realistic default across SQS, RabbitMQ, and Kafka alike.
Where this shows up in the products we build
We hit this decision inside our own products regularly, not just for clients. An IoT telemetry pipeline ingesting readings from a device fleet, where several downstream consumers (alerting, storage, a dashboard) each need the same stream independently, is a reasonable case for Kafka-shaped tooling or a managed pub/sub equivalent. A one-off background task, like generating a meeting summary after a transcription finishes, is squarely an SQS-shaped problem: one producer, one consumer type, no replay requirement. Using the same tool for both because it is already in the stack is how a simple background job ends up depending on infrastructure it never needed.
A decision checklist
If a database table with a status column and a poller would work, start there and revisit once it hurts.
If it is background jobs with one consumer type and no replay need, use a managed queue (SQS or equivalent) and spend the saved operational time elsewhere.
If you need routing rules, priorities, or fine-grained delivery control, RabbitMQ earns its operational cost.
If multiple independent consumers need the same event stream, or you need to replay history, that is the actual case for Kafka or a managed equivalent, not “it scales better.”
Whatever you pick, design consumers assuming at-least-once delivery. That assumption will still be true even if you change brokers later.
Ask who operates it day to day before asking what it can do. A tool nobody on the team can debug at 2am is a liability regardless of its feature set.