Most product teams building an AI feature default to a cloud API call without weighing the alternative. That default is often the right one. It is also, increasingly, not the only sensible option, and the cost of getting it wrong tends to stay invisible until the feature is live and either the cloud bill or the latency complaints start arriving.
The question worth asking before writing any code is not “is on-device AI better than cloud AI”. It is which parts of a given feature should run where, and why. That answer changes per feature, sometimes per user, and it is worth treating as an architecture decision rather than a default.
Four variables, not one
Four things pull in different directions depending on the feature: latency, cost at scale, where the data goes, and how capable the model needs to be. Get the weighting wrong on any one of them and the architecture that looked obvious in a demo becomes the wrong one in production.
Latency
A cloud call has a round trip built in, even before the model does any work: network out, queue, inference, network back. On a good connection that might be tens of milliseconds. On a mobile connection in a car park or a warehouse with poor signal, it can be seconds, and it will fail intermittently rather than consistently, which is harder to design around than a fixed delay. On-device inference removes the network variable entirely, at the cost of running on whatever hardware the user happens to own. For anything that needs to feel instant, live audio feedback, camera-based interaction, typing suggestions, the network round trip is often the deciding factor on its own.
Cost at scale
A cloud inference call that costs a fraction of a penny looks irrelevant in a pilot with a handful of users. Multiply it by a continuous stream, a meeting transcription running for an hour, a video feed being analysed frame by frame, a background process checking in every few seconds, and the monthly bill scales with usage in a way that is easy to underestimate at the proposal stage. On-device inference shifts that cost to the user’s hardware and battery instead of to a recurring invoice. Neither is free. The difference is who is paying, and whether the bill grows with your user base or stays flat.
Where the data goes
Sending audio, video, financial records or health-adjacent data off a device and into a third-party API is a decision with UK GDPR consequences, not just an engineering one. It means a data processing agreement, a lawful basis, a line in the privacy policy that a user might actually read, and an honest answer if a customer asks where their data is processed and by whom. Keeping inference on-device does not remove every compliance obligation, but it does remove an entire category of question about cross-border transfer and third-party processors. For features touching sensitive personal or financial data, that is often reason enough to prefer on-device processing even when the cloud option would be cheaper or more capable.
Model capability
On-device models are constrained by what will fit and run acceptably on consumer hardware: smaller, quantised, narrower in what they do well. Cloud models can be larger and more general, and can be upgraded without shipping an app update. Some tasks genuinely need that headroom, open-ended summarisation, reasoning across a long document, understanding an ambiguous instruction. Others do not: classifying a transaction into a spending category, detecting a wake word, flagging motion in a video frame are all tasks a small on-device model handles reliably. Using a frontier cloud model for a task a basic classifier would solve is a common way to overpay for capability nobody needed.
Where this actually bites
These trade-offs stop being abstract once a system is running continuously rather than answering occasional requests. A real-time transcription feature has to decide, moment to moment, whether audio is worth sending onward at all, because streaming everything to the cloud regardless of whether anyone is speaking is an easy way to pay for silence. A family organiser that listens for scheduling requests has to decide how much processing happens before anything leaves the device, both for cost and because “an app is always listening and sending audio to a server” is not a sentence that reassures anyone. A personal finance tool that categorises transactions has an easier case for on-device processing than most: the categorisation model does not need to be large, the data is sensitive by definition, and the offline case, no signal, a transaction still needs categorising, is not an edge case, it is a Tuesday. A video analytics platform has the opposite pressure: image models with any useful accuracy tend to need cloud-scale compute, so the on-device layer is usually reduced to lightweight filtering, motion detection deciding what is worth sending, rather than doing the analysis itself.
None of these land on the same answer, which is the point. The right architecture follows from what the feature actually needs, not from whichever option was easiest to prototype.
The hybrid pattern most systems land on
Pure on-device and pure cloud are the two extremes, and most production systems end up somewhere between them. A common pattern is a lightweight on-device pass that decides whether the expensive step is even needed: voice activity detection before a transcription call, motion detection before a video frame is analysed, a quick on-device classifier before an ambiguous case is escalated to a larger cloud model. The device does the cheap filtering continuously; the cloud does the expensive reasoning only when there is something worth reasoning about. This keeps the cost proportional to genuine activity rather than to wall-clock time, keeps most raw data off the network by default, and still gives access to a larger model when the task actually needs one.
The trade-off is complexity: two execution environments to build, test and maintain instead of one, and a synchronisation problem between them when the device is offline and then reconnects. That complexity is worth paying for when the four variables above genuinely disagree with each other. It is not worth paying for on a feature where cloud alone already satisfies latency, cost and data handling requirements.
Questions worth asking before committing
- What latency does the feature need, tested on a realistic mobile connection, not office wifi?
- Is offline a rare edge case for this feature, or is it the primary condition it needs to work in?
- What data would leave the device, and could you explain that plainly to a regulator or a customer who asked?
- What does the cost look like at ten times current usage, not at pilot scale?
- Does the task need a large, general model, or would a small, narrow one do the job at a fraction of the cost?
- Does the choice lock you into a specific cloud vendor or device capability you might want to change later?
If the answers point in different directions for different parts of the feature, that is not a sign the architecture is wrong. It is usually a sign the feature needs both, split along the line where each does its own job best.