A dashboard is easy to commission and hard to trust. The build itself — connecting sources, modelling the data, drawing the charts — is the visible part of the work, and it’s usually what gets scoped, quoted and signed off. Whether the numbers on that dashboard are actually right is a separate question, and it’s the one that gets asked far too late: after launch, when someone in a meeting says “that figure doesn’t look right” and nobody can say with confidence whether it’s wrong, or just surprising.
Data quality isn’t a QA step you bolt on at the end. It’s a set of decisions about what “correct” means for your business, made before a single pipeline is built. If those decisions aren’t made explicitly, they get made implicitly, by whichever assumptions happen to be baked into the source systems.
Why it gets skipped in scoping
Most analytics projects are scoped around outputs: which dashboards, which metrics, which sources connected. That’s reasonable — it’s what a client can picture and what a supplier can quote against. Data quality work doesn’t produce a chart, so it’s easy to leave implicit, or to assume it’s covered by “we’ll validate as we go.”
The problem is that “as we go” usually means the first time a wrong number gets noticed, which is after launch, in front of the people the dashboard was built to inform. By then the fix isn’t a validation rule, it’s a trust repair exercise, and those take a lot longer.
The cheaper point to catch this is before the build starts, while you can still ask the source systems to change, not just the pipeline reading from them.
Where does each number actually come from
For every metric that matters — revenue, active users, conversion rate, whatever it is for your business — trace it back to the field or fields it’s computed from, and ask who enters that data and when.
A “completed order” in your operations database might mean something different from a “completed order” your finance team reports on, because one includes cancelled-and-refunded orders in the count for a few hours before the refund posts, and the other doesn’t. Neither system is wrong. They’re answering different questions. An analytics project that pulls from one without checking the other’s assumptions will produce numbers that are individually defensible and collectively inconsistent.
This is slower than it sounds, because the people who can answer it are usually not the people commissioning the dashboard. It requires going to whoever owns the source system and asking, specifically, what a given field means, what populates it, and what can make it null, duplicated, or stale.
What happens when a field is missing or wrong shape
Every source system has records with missing fields, malformed values, or entries that arrived out of order. The question worth asking before the build starts is not whether this will happen — it will — but what the pipeline should do about it.
There are three honest options, and a project should pick one deliberately for each critical field, rather than defaulting to whichever the tooling does automatically:
- Reject the record and flag it, accepting that some data won’t reach the dashboard until it’s fixed at source.
- Accept the record with a default value, accepting that the metric will be slightly wrong in a known, bounded way.
- Halt the pipeline and alert someone, accepting that the dashboard will be late rather than wrong.
Most teams want the third option for financial metrics and the first for anything lower stakes, but that’s a judgement call specific to the business, not something a generic pipeline template will get right by default.
Who owns the definition of each metric
Ambiguity in a metric’s definition is not a technical problem, and no amount of pipeline engineering fixes it. If “active user” means something different to the product team than it does to the board, the fix is a conversation and a written definition, not a query change.
Before committing to a build, it’s worth writing down, in plain language, the definition of every metric that will drive a decision — not the SQL, the sentence. If two stakeholders read that sentence and picture different things, that gap needs closing before the dashboard exists, because once it exists, the numbers acquire an authority that’s hard to walk back.
How will you know if the numbers are wrong, later
A dashboard that was correct on launch day can go wrong quietly: a source system changes a field’s meaning, an integration silently stops receiving updates, a schema change upstream shifts a column’s units. None of this throws an error. It just makes the dashboard wrong in a way nobody notices until the totals stop making sense against something else.
The practical defence is a small number of automated checks that run continuously, not a one-off validation at handover:
- Row counts and freshness: has each source delivered data on the schedule it’s supposed to, and does the volume look like a normal day rather than a tenth of one?
- Range and null checks on the handful of fields that actually drive decisions, not every field in the schema.
- A reconciliation point against an independent source — finance totals against the payment processor’s own reports, for instance — checked on a schedule, not just at launch.
Five checks that actually run and get looked at are worth more than fifty that get set up once and ignored. The question to ask a supplier isn’t “will you validate the data” — everyone says yes to that — it’s “which specific checks will run, on which fields, and who sees the alert when one fails.”
The trade-off: you can’t validate everything
Comprehensive validation on every field, from every source, checked continuously, is achievable but expensive, and most businesses don’t need it for most of their data. The judgement to make explicitly, at scoping stage, is which metrics are load-bearing — the ones a real decision gets made from — and concentrate the validation effort there. A dashboard showing page views for internal curiosity can tolerate more slack than the one finance uses to reconcile revenue.
This is also where a discovery phase earns its cost: identifying which handful of numbers actually matter is a business conversation, not a data engineering task, and it has to happen before the validation rules are written, not after.
A checklist before you sign off the scope
- For each metric on the dashboard, can you point to the exact field or fields it comes from, and name who owns that source system?
- Is there a plain-language definition of each key metric that two different stakeholders would read the same way?
- For each critical field, has someone decided, deliberately, whether a bad value gets rejected, defaulted, or halts the pipeline?
- Are there automated freshness and range checks on the load-bearing metrics, not just a one-off check at launch?
- Is there at least one reconciliation point against an independent source for the numbers that drive real decisions?
- Does someone specific see the alert when a check fails, and is there an agreed response?
If you can’t answer most of these before the build starts, the project isn’t under-scoped on features — it’s under-scoped on trust, and that tends to surface at the worst possible time.