Database recovery

TL;DR

I intended to delete one company’s database records and instead deleted every row in a shared table. We recovered the missing data from a dump, company by company, rather than restoring the whole database and losing unrelated changes.

Context

One of the worst production mistakes I have made was trying to delete one company’s database records and instead deleting every row in the table.

This was in a shared multi-tenant database, which made the recovery decision much more important than the initial mistake.

At first I thought I had probably used the wrong company_id. That would have been bad, but at least contained. A day later, more customers reported missing data, and the real failure mode became obvious: I had likely executed the delete without the intended WHERE clause.

Recovery decision

We needed to recover the deleted records without discarding other customers’ changes.

Restoring the previous backup would have been faster, but it also would have wiped out a full day of legitimate changes for everyone else.

So we took the harder path. We tested the recovery locally first, rebuilt the deleted data from a dump, and reconstructed the entire relational data set company by company.

That let us recover the missing customer data without rolling back unrelated production changes.

What this account establishes

We tested the recovery locally and documented the procedure. The scope of the deletion became apparent a day later, after more customers reported missing data.

This account does not include the exact recovery duration, a verified data-loss total or the safeguards introduced afterward. It explains the recovery decision, but it is not a complete postmortem or evidence that recurrence was prevented.