Most conversations about data retention start with a compliance question: how long are we allowed to keep this? That question has an answer, but it is rarely the hard part. The hard part is that a retention policy, once you take it seriously, turns into a set of engineering decisions that touch your database schema, your backup strategy, your analytics pipeline and your logging setup. Written down in a policy document, retention looks like a paragraph. Implemented properly, it is a project.
The legal starting point is simpler than the engineering problem
Under UK GDPR, the storage limitation principle says you should not keep personal data for longer than you need it for the purpose you collected it. There is no universal number attached to that rule. Some sectors have fixed minimums set by other legislation (financial records for HMRC, anti-money laundering checks, professional indemnity periods), but for most SaaS products and internal business systems, you are expected to set your own period and be able to justify it.
That is the easy part to write down. A line like “we delete inactive user accounts after 24 months” fits neatly into a privacy policy. Making that true across every system that touched that user’s data is where most companies quietly give up.
Retention is a schema decision, not an afterthought
If deletion is bolted on after a product has been running for a couple of years, you usually find the same problems: foreign keys that assume a user row always exists, audit trails that reference IDs directly instead of a snapshot, and reporting tables that join back to live customer data rather than storing what they need at the time. None of this is wrong when you build it – it is the natural way to build a relational schema quickly. It just makes deletion expensive later.
Two decisions worth making early, even if retention feels a long way off:
- Soft delete or hard delete, per table, not as a blanket rule. Soft deletes (a
deleted_atcolumn) are cheap and reversible, but they are not deletion – the data is still there, still in backups, still readable by anyone with database access. Treat soft delete as a staging step before a scheduled hard purge, not as the end state. - Decide what survives deletion on purpose. Financial records, dispute evidence and security logs often need to outlive the account they relate to. Design for that explicitly – store the minimum needed (an anonymised transaction record, not a full customer profile) rather than leaving the full row in place because deleting it felt risky.
Backups do not obey your retention policy by default
This is the gap we see most often. A team deletes a user from the production database, and considers the job done. Meanwhile, thirty days of nightly backups still contain that row, and depending on the backup tooling, so might every backup for the following year if retention on the backup system was configured once, years ago, and never revisited.
You do not usually need to scrub deleted records out of every historical backup – that is often disproportionate, and most privacy frameworks accept that backups have their own, shorter retention cycle rather than being purged on demand. What you do need is for someone to have actually decided that, written it down, and set the backup retention window to match. “We didn’t think about it” is a different position to defend than “we decided backups roll off after 35 days and that is documented.”
Logs and event data are a separate problem, and usually the bigger one
Application logs, analytics events and IoT telemetry tend to accumulate by default rather than by decision. Event pipelines are usually built to answer “can we capture this?” not “how long do we need it?”, and the answer to capture is almost always yes, because storage is cheap and nobody wants to be the person who deleted the row that would have explained an incident six months later.
The result is data warehouses and event stores holding raw, identifiable events indefinitely, often including data nobody downstream is actually querying past the first few weeks. This is where anonymisation earns its place: if a dashboard needs trends, not individuals, aggregate or pseudonymise the data after a defined window instead of keeping the raw event forever. Pseudonymisation (replacing identifiers with tokens you could theoretically reverse) and anonymisation (removing the link entirely) are different things with different obligations attached – worth getting a straight answer on which one a given pipeline actually does, rather than assuming.
Where the cost actually sits
Storage itself is rarely the expensive part of this. The cost is in three other places:
- Engineering time to build and test deletion paths that correctly cascade across every table, cache, search index and downstream system that holds a copy of the data – and to prove they worked, not just assume they did.
- Query and index performance on tables that grow without bound. A five-year-old events table with no partitioning or archiving strategy slows down every query that touches it, whether or not any individual row is still relevant.
- Third-party systems you don’t control directly – the CRM, the email platform, the support desk, the analytics tool. Each one holds a copy of some subset of your data, on its own retention terms, and someone needs to know that inventory exists.
Treat the policy as a specification, not a document
A retention policy that only exists as prose in a privacy policy is not enforceable by anyone except a human who remembers to act on it. A policy that is useful to an engineering team looks more like a table: data category, system it lives in, retention period, deletion mechanism, and how you would verify it actually ran. That table is the actual deliverable. The privacy policy wording is a summary of it, not the other way round.
If you can’t point to the job, script or process that deletes a given category of data, you don’t have a retention policy for it – you have an intention.
Questions worth answering before you commission or extend a system
- For each category of personal or sensitive data, is there an agreed retention period, and is it written somewhere other than a privacy policy?
- Does deletion in the primary database actually cascade to caches, search indexes, analytics pipelines and third-party tools, or does it stop at the first table?
- Is backup retention a deliberate, documented setting, or whatever the default was when the backup system was first configured?
- Do your event and log pipelines have an archiving or anonymisation step, or do they grow indefinitely by default?
- If an auditor or a customer asked you to prove a specific person’s data was deleted eighteen months ago, could you show the mechanism that did it?
If most of those have a clear answer, retention is under control. If they don’t, it is worth treating as a discrete piece of work rather than something to patch in later – it is considerably cheaper to design deletion into a schema than to retrofit it once several years of data and a dozen integrations depend on the current shape.