Most small engineering teams handle observability in one of two ways. Either they run with nothing beyond a hosting provider’s default dashboard and find out about problems from a customer email, or they sign up for a full observability platform in month one, wire up every agent the documentation suggests, and stop looking at any of it within a quarter. Both outcomes come from the same mistake: treating observability as a product to install rather than a set of questions you need answered.
If you are commissioning or building a product and wondering what monitoring actually needs to exist before launch, the useful exercise is not “which tool” but “which question, in which order”. This is how we think about it when we scope a new system, whether it is a customer-facing SaaS product or a fleet of IoT devices reporting telemetry.
Logs, metrics and traces answer different questions
The three signals get bundled together under “observability” so often that teams forget they exist to answer different things.
- Metrics tell you something is wrong: error rate has climbed, latency has doubled, queue depth is growing. They are cheap to store, cheap to alert on, and deliberately lossy — you get a number, not a story.
- Logs tell you what happened in a specific request or job: which user, which input, which line failed. They are expensive at volume and mostly useless unless they are structured and searchable.
- Traces tell you where time went across a request that crosses multiple services: which downstream call was slow, which step in the pipeline stalled. They matter once you have more than one service in the request path, and not much before that.
A team of three engineers running a monolith against one database rarely needs distributed tracing. A team running four microservices behind an API gateway, calling two third-party APIs per request, will struggle to debug a latency spike without it. Match the signal to the actual shape of the system, not to what a vendor’s pricing page implies you should be running.
Start with the critical path, not full coverage
Before anything else, instrument the handful of operations that, if they broke silently, would cost you customers or money. For most products commissioned by small businesses that is a short list: sign-in, the core action the product exists to perform (booking a shift, recording a transaction, starting a transcription), and payment or billing if it applies. Everything else — admin screens, internal reporting, rarely-used settings pages — can wait.
For each of those operations, you want three things before launch: an error rate you can see on a graph, a latency figure for the slow tail (the 95th or 99th percentile, not the average, since averages hide the requests that actually annoy people), and a way to see the specific failed request when the graph tells you something broke. That third one is what structured logging is for — a log line with a request ID, a user ID, and the exception, not a paragraph of prose written for a human reading a terminal in real time.
Alerting is a design problem, not a checkbox
The fastest way to make monitoring worthless is to alert on everything that could theoretically indicate a problem. A team that gets paged for CPU spikes, deploy events, and single failed requests learns within a month to mute the channel, and then misses the alert that actually mattered.
The discipline that holds up is alerting on customer-visible symptoms, not internal causes. “Error rate on the checkout endpoint exceeded 2% for five minutes” is a good page. “CPU on server 3 is at 80%” is not, unless you already know that specific threshold reliably predicts a customer-facing problem — and most teams that add it have not checked. A rough version of the same rule:
alert: HighErrorRate
condition: error_rate(endpoint="/checkout") > 2% for 5m
action: page on-call
severity: high
alert: ElevatedCPU
condition: cpu_usage(host="app-3") > 80% for 10m
action: log only, review weeklyCause-level metrics like CPU, memory and queue depth are still worth collecting — they help you diagnose an incident once you know one is happening. They are just rarely worth waking someone up over on their own.
Self-hosted or SaaS is a cost and time question, not a technical one
Prometheus and Grafana, or the Elastic stack, cost nothing in licence fees and a real amount in engineering time: someone has to run them, upgrade them, and tune retention so the disk does not fill up. A hosted platform costs money from day one but removes that maintenance burden, and most price on data volume — so the bill grows with the exact thing you are trying to get more of.
For a small team, the honest trade-off is usually: self-hosted if you already run infrastructure you are comfortable operating and have the spare time to maintain another service; hosted if your engineering time is better spent on the product than on running a metrics stack. Neither choice is a mistake. What is a mistake is picking a hosted platform’s most expensive tier for the extra features and then using a tenth of what you are paying for, or self-hosting a stack nobody has budgeted time to patch.
What to leave until you actually need it
It is worth being explicit about what does not belong in a first pass, because most of the cost overruns we see in monitoring come from instrumenting things nobody ends up looking at.
- Continuous profiling — useful once you have a specific, recurring performance problem you cannot explain from traces alone, not before.
- Distributed tracing across a system with one or two services — the request path is short enough that logs and metrics already tell the story.
- Log retention beyond what compliance or debugging actually requires — most incidents are diagnosed within days, not months, and long retention mainly adds cost and, for personal data, a GDPR question you now have to answer.
- Dashboards built for stakeholders rather than for the engineer on call — nice to have, but they do not stop an incident and are usually the first thing to go stale.
A checklist before you call it done
Before treating observability as “in place” for a new product, it is worth being able to answer yes to each of these:
- Can you see error rate and slow-tail latency for every operation that would cost you a customer if it broke?
- Can you go from an alert to the specific failed request, with enough context to reproduce it, in under five minutes?
- Does every alert that pages someone correspond to something a customer would notice — and has that link actually been tested, not assumed?
- Do you know, roughly, what a doubling of traffic would do to your monitoring bill — and is that acceptable?
- If the person who set this up left tomorrow, could someone else read the alert rules and understand why each one exists?
If the answer to any of those is no, that is the gap worth closing next — not the next tool on the market.