A backup you have never restored is just a file

A routine review of our own infrastructure turned up something embarrassing: one product had no backups at all. Not stale backups. None. The application next to it on the same machine had nightly database dumps with two weeks of retention, and had done for a year.

Nobody decided this. The product had grown from a weekend prototype into something people used every day, and the moment where you would sit down and set up backups never arrived, because there is no such moment. It is always slightly less urgent than the feature you are working on.

Writing the backup script took an afternoon. Convincing ourselves it worked took considerably longer, and that is the part worth writing down.

Verifying the gzip is not verifying the backup

The first version checked the dump with gzip -t. That confirms the archive is not corrupt, which felt like enough.

It is not. A database dump that dies halfway through — because the disk filled, or the process was killed, or the connection dropped — produces a perfectly valid gzip of a truncated file. The integrity check passes. The archive decompresses cleanly. It just does not contain your data.

So the check became a size floor as well: below a threshold, the backup is treated as failed and the job exits non-zero. Crude, and it catches the actual failure mode.

Restoring it is the only test that counts

We restored a dump into a scratch database and compared the row counts of every significant table against the live system. They matched exactly.

That exercise is the difference between having a backup and believing you have one. Everything up to that point — the script runs, the file appears, the size looks right — is circumstantial. The restore is the evidence.

It also has to be repeated. A backup that restored correctly before a schema change tells you nothing about one taken after it. Our rule now is that a schema migration is not finished until the restore drill has been run against the new shape.

Two machines, each holding the other's data

Backups on the same machine as the data protect you against one thing: someone deleting a row. They protect against nothing else — not disk failure, not a lost machine, not an account problem.

So the topology is crossed. The database lives on one machine and its dumps are pushed to a second. The large media files live on the second machine and are mirrored back to the first. Each machine holds the other's data, and losing either one loses nothing.

Three details in that mirror are deliberate, and all three look like mistakes if you do not know why:

• The mirror does not delete. A file removed from the source stays in the mirror. This means the mirror grows and occasionally holds things nobody wants any more, and it means an accidental mass deletion cannot propagate. Surviving an accidental deletion is most of the reason a media backup exists.
• It refuses to run when the source is not mounted. If the storage fails to mount, the source directory is not empty-looking, it is empty — and a faithful mirror would then replicate emptiness over the only copy. The job checks it is looking at a real mount and aborts otherwise.
• The schedule avoids the neighbours. The job runs at an odd minute of an odd hour, chosen because the other scheduled work on that machine runs on the hour. Two heavy jobs at 03:00 are not two independent jobs.

What we still have not solved

Both scripts write logs and exit non-zero when something is wrong. Nothing pages a human.

If backups silently stop tonight, we find out the next time someone reads a log file. For a product people rely on, that is not good enough, and writing this up is partly an admission that the alerting is still missing.

There is also no restore drill for the media files, only for the database. We have proved we can bring the rows back. We have not proved, end to end, that we can bring the photographs back — and until that has been done once, it belongs in the same category as the backup we thought we had.

The short version

1. Check the size, not just the integrity — a truncated dump is a valid archive.
2. Restore it, or you do not know. Re-run that drill after every schema change.
3. Put the copy on a different machine from the original.
4. Never let a mirror delete, and never let it run against a store that failed to mount.