Streaming architectures are chosen more often than they are needed. The reasoning usually runs: our data is important, important data should be current, therefore we need real time. Each step sounds fine and the conclusion commits a mid-sized company to a system that costs several times more to run and requires skills it does not have on staff. The question worth answering first is not how fresh the data can be, but how fresh it has to be before anything changes.
Latency is a business requirement
The useful way to frame freshness is to trace what happens to the data after it lands. Every pipeline exists to support a decision or an action. Work out who or what takes that action and how quickly they can take it, and the latency requirement writes itself.
Some examples of the reasoning:
- A finance director reviews margin by product line and adjusts pricing at the start of each month. Data that is a day old is indistinguishable from data that is a second old. Nightly is correct.
- An operations manager reallocates staff across sites during the working day. Hourly changes the decision; minute-by-minute does not, because the reallocation itself takes longer than that.
- A fraud check blocks a transaction before it completes. The action is automated and happens in the request path. This is genuinely real time, and it is not a pipeline problem at all — it belongs in the application.
- An IoT deployment monitors equipment for fault conditions. If a fault requires an engineer, response time is measured in hours; if it triggers an automatic shutdown, in seconds. The same telemetry, two different architectures, decided by what happens at the other end.
The pattern is consistent: latency requirements come from the speed of the response, not the speed of the data. A dashboard refreshed every ten seconds and looked at twice a day is expensive theatre.
One caveat. Sometimes the request is not “we need fresh data” but “we need it to arrive without someone pressing a button” — the real complaint is manual work, not latency. A well-built scheduled pipeline solves that completely.
What each option actually costs
Compute is the smallest part of the difference, which is why comparisons based on cloud pricing pages mislead.
A batch pipeline runs on a schedule and is idle the rest of the time. It runs on ephemeral compute, so you pay for the hour it takes rather than for the day. When it fails, it fails at three in the morning, an alert fires, someone fixes the cause and reruns the job, and by nine o’clock the data is correct with nobody outside the team aware anything happened. That property — a rerun makes everything right — is the single largest operational advantage of batch, and it is easy to undervalue until you have lived without it.
A streaming pipeline is a permanently running distributed system. The costs it brings are:
- Always-on infrastructure. Brokers, stream processors and their state stores run continuously, sized for peak throughput rather than average, and they cost the same at three in the morning as at midday.
- Incident response as a standing commitment. A stalled consumer accumulates lag that grows for as long as nobody notices. Recovery is not a rerun; it is catching up while new data continues to arrive, on infrastructure sized for normal load.
- Correction is genuinely hard. Reprocessing after a bug means replaying from a retained log, dealing with downstream systems that already consumed the wrong values, and managing the fact that some of those consumers sent emails or updated ledgers. In batch you rerun yesterday. In streaming you plan a backfill.
- Correctness problems that do not exist in batch. Events arrive out of order and late. Deciding when a time window is closed, what to do with an event that arrives after you have already reported on that window, and how to make processing idempotent so a redelivered event does not double-count — these are unavoidable design work in streaming and mostly free in batch, where the boundary is simply “yesterday”.
Then there is staffing, which is where mid-sized companies get caught. Batch pipelines can be maintained by people who know SQL and a scheduler — a description that fits most analytics teams and many capable developers. Streaming needs someone comfortable with distributed systems failure modes, consumer group mechanics, state management and watermarking. That person is harder to hire, more expensive, and if there is only one of them, the pipeline has a single point of failure that takes annual leave.
The honest question to put to a team proposing streaming: who is on call for it, and what happens when they leave?
Micro-batch: the option most companies want
The choice is not binary, and the middle ground covers most real requirements. Micro-batch means running the same batch logic frequently — every five or fifteen minutes rather than nightly — processing only what has changed since the last run.
What this preserves is the operational model. Each run is discrete, has a defined input, either succeeds or fails, and can be rerun. You keep the debuggability and the ability to correct mistakes by rerunning, while delivering data fresh enough for essentially every human-in-the-loop decision. The tooling is ordinary: a scheduler, incremental extraction based on a watermark column, and a warehouse that can handle a merge.
The things to get right are unglamorous but they are the whole job. Extraction must be genuinely incremental, or you are running a full reload every fifteen minutes and the cost curve turns against you. Loads must be idempotent — a merge on a business key rather than an append — so a rerun after a partial failure does not duplicate rows. Runs must not overlap; if a run takes longer than the interval, the scheduler needs to skip rather than start a second copy. And late-arriving data needs a policy: usually a periodic reprocessing window that revisits the last few days and corrects anything that shifted.
Change data capture belongs in the same family. Reading a database’s transaction log rather than polling a timestamp column gives near-real-time freshness, catches deletes and updates that polling misses, and puts far less load on the source. For keeping a warehouse aligned with an operational database it is frequently the right answer.
When streaming is the right call
There are real cases, and they share a characteristic: something automated acts on each event, individually, within seconds, and the value of acting decays fast.
- Detecting and blocking fraudulent activity as it happens.
- Safety-critical or availability-critical monitoring where a machine responds to a threshold breach without waiting for a person.
- Products whose core function is live — collaborative editing, live transcription, real-time positioning.
- Event volumes where batching would mean moving impractically large quantities at once, and a continuous flow is simply the more efficient shape.
Note what is absent from that list: dashboards, reporting, most machine learning feature pipelines, and anything whose consumer is a person who looks at it a few times a day.
A path that does not trap you
Sequencing matters as much as the choice. Start with the schedule the business decision actually requires, which for most reporting is nightly. Build the logic so it is incremental from the beginning even if it runs once a day, because retrofitting that later means rewriting it. Instrument freshness, so that “the data is too old” becomes a measured claim rather than an impression. When a specific decision demonstrably suffers from the delay, shorten the interval for that pipeline alone — not for everything.
Most pipelines never need to go further. The ones that do will announce themselves clearly, with a named decision, a named owner and a cost of delay someone can put a figure on. Until that case exists, the cheaper architecture is also the more reliable one, and reliability is what people actually mean when they say they want better data.
Four questions before you choose
- What decision or action does this data drive, and how quickly can whoever takes it respond?
- What does being an hour out of date cost, in terms someone in the business would recognise?
- If the pipeline produced wrong results for two days, how would we correct them?
- Who operates this at three in the morning, and who operates it when that person leaves?
If questions three and four have vague answers, the architecture is too complicated for the organisation, whatever the latency requirement says.