Ask most operations teams running a connected fleet what they think of their alerting, and the honest answer is usually some version of “we’ve stopped reading it.” Not because the sensors are wrong. Because someone, at some point, wired every telemetry event straight into a notification, and now the channel is too noisy to trust. The fix people reach for first — adding more thresholds — almost always makes it worse.
This comes up a lot when we work on IoT platforms for fleet telemetry: devices reporting position, battery, temperature, connectivity, usage cycles. The hardware and the data pipeline are rarely the hard part. Deciding what deserves to interrupt a person, and how, is.
Telemetry is not an alert
The first design mistake is treating every measurement as a candidate notification. A device reports battery level every five minutes; that’s telemetry. Battery level has crossed a threshold that predicts failure within 48 hours; that might be an alert. The two get conflated because it’s technically simple to fire a webhook on any value change, and technically harder to decide which value changes matter.
Separating them early pays off. Telemetry goes to storage and dashboards, where someone can query it when they choose to look. Alerts go through a separate pipeline with its own rules for severity, deduplication and routing. If a device team can’t explain why a given event is in the alert pipeline rather than just the dashboard, it probably shouldn’t be there.
Where the fatigue actually comes from
Three patterns account for most of the noise we see in fleet alerting:
- Fixed thresholds applied to a variable population. A single temperature ceiling applied to every unit ignores that hardware, location and duty cycle differ across a fleet. What’s normal for a device running outdoors in summer is an anomaly for one in a server room.
- No hysteresis. A value that hovers right at a threshold triggers repeated alerts as it crosses back and forth, sometimes dozens of times an hour for the same underlying condition.
- Flat severity. When a dead battery and a device going fully offline arrive in the same Slack channel with the same formatting, people stop distinguishing them, and eventually stop reading either.
- Baseline per device, not per fleet. Let each unit establish a normal range over its first weeks of operation, and alert on deviation from its own baseline rather than a single fleet-wide number. This is more engineering work up front and considerably less noise afterwards.
- Use bands, not lines. A threshold with a buffer zone — enter alert state at one value, clear it only after dropping back past a lower value — stops a borderline reading from firing repeatedly.
- Add a cooldown window. Once an alert has fired for a device, suppress repeats of the same condition for a defined period. The person on the receiving end needs to know something changed, not that the sensor is still reading the same number it read ten minutes ago.
- Tier severity deliberately. A device going offline mid-shift is not the same class of event as a firmware version drifting from the fleet baseline. Give them different channels, different formatting, and different response expectations, and say so explicitly in the alert itself.
- Can you name, for any alert type currently firing, why it’s an alert and not just a dashboard metric?
- Do thresholds account for per-device variation, or is one number applied across a mixed fleet?
- Is there a cooldown or hysteresis band preventing repeat fires on borderline readings?
- Do different severities route to genuinely different places, with different response expectations?
- When was the last time someone audited which alerts get acknowledged versus muted?
- If the answer to that last one is “never,” that’s the place to start — not with new sensors, and not with more thresholds.
None of these are exotic problems. They’re what happens when alerting is bolted onto a telemetry pipeline after the fact, rather than designed as its own system.
Thresholds that hold
A few design choices tend to make the biggest difference in practice:
None of this requires machine learning or an anomaly-detection product. Baselining, bands and cooldowns are ordinary state-machine logic. The judgement is in deciding what belongs in each tier for a given fleet, which depends on what the operations team actually needs to act on — a question worth answering with them directly, not assuming from the data model.
Routing is a decision, not an afterthought
Where an alert goes matters as much as whether it fires. A critical, safety-relevant alert probably needs paging and an on-call rotation. A gradual battery-degradation trend across several units is useful information for a weekly review, not a 2am page. Routing every severity to the same Slack channel is the fastest way to make the whole channel ignorable, and once a team has learned to mute a channel, restoring trust in it takes far longer than building it did.
It also helps to separate the alert from the investigation. The notification should say what happened and where; it shouldn’t require someone to open three dashboards to work out whether it’s worth acting on. Linking directly to the relevant device history from the alert itself removes a step that, multiplied across hundreds of alerts a month, adds up to real lost time.
Build versus configure
Generic IoT platforms usually offer threshold alerting out of the box, and for a small fleet with simple failure modes, that can be enough. The case for custom alerting logic shows up when a fleet has heterogeneous hardware, domain-specific failure patterns, or when alerts need to feed into existing operational tooling — a ticketing system, a bespoke ops dashboard, a compliance log. At that point the question stops being “can the platform send a webhook” and becomes “does the logic deciding when to send it actually match how this fleet fails,” which off-the-shelf thresholds rarely do without significant configuration work anyway.
Whichever route a team takes, the underlying design questions are the same ones above. Buying a platform doesn’t remove the need to answer them; it just changes where the configuration lives.
A checklist before you trust your alerting
Worth running through this with whoever owns the on-call rotation, not just the engineering team: