A bad web deploy is embarrassing for ten minutes. A bad firmware deploy can put a thousand devices permanently beyond the reach of any further deploy, in locations needing a van and a ladder. That asymmetry should drive every decision in an over-the-air update system. The goal is not fast updates; it is that no update, however broken, can remove your ability to send the next one.
Atomicity is the whole game
The failure mode that ends fleets is the partially written image. Power drops, or the link dies, midway through overwriting the running system. What is left on flash is neither the old firmware nor the new one, and no code remains that knows how to fetch a replacement.
The standard answer is dual-bank, or A/B, partitioning. The device carries two full system slots. It runs from slot A while writing the new image into slot B, verifies that image once it is completely written, and only then flips a boot flag. The flip is a single small write — the one operation that must be atomic, and the only one that is. Lose power before the flip and the device reboots into A unchanged; lose it after and the image in B is already complete and verified.
The cost is roughly twice the flash for the system image, which pushes the bill of materials up. Where that is not affordable, the fallback is a small immutable recovery partition whose only job is to fetch a fresh image when the main slot fails to boot. What is not acceptable is a single slot with no recovery path.
Rollback has to be automatic, and it has to be dumb
Booting the new image is not the same as the new image working. A device can boot cleanly and then fail to bring up its radio, crash-loop three minutes in, or come up fine and never talk to the backend again. Rollback logic that depends on the new firmware diagnosing itself is circular reasoning.
The mechanism that works is a boot-confirmation watchdog held outside the application. The bootloader marks the new slot as trial, increments a counter, and boots it. The new firmware must explicitly confirm itself by writing a flag — after doing real work: network up, backend reachable, sensors reading. If the device reboots without that flag, or the counter exceeds its limit, the bootloader reverts to the previous slot without consulting anything the new firmware might have corrupted.
Two details matter. Confirmation criteria should include end-to-end connectivity, not just local health, because a common OTA regression is a device that runs perfectly and no longer phones home. And the trial period must be long enough to catch slow failures — a memory leak that kills the device after six hours is invisible to a sixty-second watchdog.
Telemetry ships before the update does
This is the ordering constraint people get wrong most often. You cannot run a staged rollout if you cannot tell a healthy device from a sick one. Without health data, “the rollout looks fine” means “no customer has rung us yet”, which is not a signal — it is a delay.
Before the first OTA-capable release goes out, devices should already report firmware version, uptime and reboot count, reboot reason, free memory and flash wear, connectivity quality, and time since last successful backend contact. Reboot count and reason are the two that detect a bad update; the rest tell you why. Telemetry is a prerequisite for the update pipeline, not a companion feature — a fleet that can receive updates but cannot report on them is more dangerous than one that cannot be updated at all, because you will use the capability blindly.
Staged rollouts and what a canary is for
Push to everything at once and a defect affecting one hardware revision, one firmware baseline or one mobile operator becomes a full fleet outage. Staged rollout limits blast radius, and is straightforward once telemetry exists.
- Internal devices first — your own bench and field units, where you can attach a debugger.
- A canary cohort chosen for diversity, not convenience: spread across hardware revisions, previous firmware versions, carriers and geographies. A hundred identical devices in one warehouse tests almost nothing.
- Progressive expansion, each stage held long enough to cover the slow-failure window, with automatic halt on a defined health regression.
- A hard stop switch that suspends the rollout server-side, so devices that have not yet fetched simply never do.
Define the halt condition before the rollout, as a rule the system enforces itself: if the check-in rate for the updated cohort falls below the baseline cohort by a stated margin, or reboot rate rises above it, stop and page a human. Left to judgement in the moment, staged rollouts get waved through.
Devices that are asleep, roaming or nearly flat
Fleet software written on a desk assumes devices are online. Real fleets are full of units on a metered cellular tariff, on a solar panel in February, or behind a gateway that wakes hourly. The update system has to treat “not now” as a normal answer.
Make the device the decision-maker. It applies its own preconditions before starting a download: battery above a threshold with margin for the write and the reboot, or on external power; a link type it is willing to spend on; enough time in its duty cycle to finish. A device that refuses an update for a week because it is on battery is behaving correctly. One that accepts at eight per cent charge and dies mid-write is a site visit. Add jitter to scheduling too: ten thousand devices told to update at midnight will produce a self-inflicted denial of service against your own distribution endpoint.
Downloads must be resumable at byte-range granularity and checksummed per chunk, because intermittent links drop mid-transfer as a matter of routine. Delta updates cut transfer volume on metered connections, but a delta is only valid against one exact source version, so the server must track per-device versions precisely and fall back to a full image whenever the baseline is not what it expects. Keep the full image path tested — it is what you will use when a fleet has drifted across a dozen versions.
Signing, and what the device must refuse
An update channel is a remote code execution channel you built deliberately. Transport security is not sufficient: TLS tells the device it is talking to your server, not that the bytes are firmware you authorised. Images must be signed, and verified against a public key in read-only storage or a secure element before the boot flag is flipped — not after.
Beyond a valid signature, the device should reject an image older than what it runs unless a rollback is separately authorised, so an attacker cannot replay a genuine but vulnerable old release, and reject an image not addressed to its hardware model and revision. The signing key belongs in a hardware security module with a documented list of who can trigger a signature: a key sitting on a build server is one compromised CI pipeline away from being the attacker’s key.
Plan key rotation while the fleet is young. Devices that trust exactly one key, burned in at manufacture, with no path to trust a successor, have a hard expiry date attached to that key’s lifetime. Provisioning two or more trusted keys from the start costs almost nothing and is close to impossible to retrofit.
A short readiness check
Before the first OTA release reaches a device you do not physically own, you should be able to answer all of these:
- If power is cut at the worst possible instant during a write, does the device still boot?
- If the new firmware boots but never reaches the backend, does it revert without human intervention, and how long does that take?
- Which specific metric tells you a rollout is going wrong, and what threshold halts it automatically?
- What does a device do when asked to update at low battery or on a metered link?
- Who can sign an image, where does the key live, and what happens when it needs replacing?
Test the first two by doing them. Pull the power during a write, on real hardware, repeatedly and at random points. Ship a deliberately broken build to a lab device and watch whether it comes home on its own. An untested rollback path is a rollback path that does not exist.