Website backups only count if you can restore them

How GitLab discovered its backups were not working

GitLab's own post-mortem, published ten days later, is unusually candid. The engineer had been rebuilding a spare copy of the database that had fallen out of step with the live one, and ran the delete command on what they believed was the spare. It was the live server.

Then the safety nets failed one after another. The daily database backup had been failing because it used an older version of the backup tool than the database required, so the storage bucket meant to hold the copies was empty. The job did send failure warnings by email, but the receiving mail server rejected them, so nobody knew. Disk snapshots, another possible layer, were not switched on for the database servers because the team assumed the other backups were enough.

The copy that saved them was never meant to be a backup: a snapshot an engineer had taken about six hours earlier to load into a test environment. Restoring it meant losing the database changes users had made since, which GitLab estimated at roughly 5,000 projects, 5,000 comments and 700 new user accounts. Copying the data back from slow storage took around 18 hours.

None of this needed exotic technology. A backup job that fails silently, a copy nobody has opened and no one responsible for checking: a five-page website can have all three.

How much work can you lose, and for how long?

Disaster-recovery planners use two ideas worth borrowing. The recovery point is how far back a restore takes you, which decides how much recent work disappears; the US National Institute of Standards and Technology (NIST) defines it as the point in time to which data must be recovered after an outage. The recovery time is how long your site can be down before the damage becomes serious.

Put them in your own terms. If your backup runs at 2am and the site fails at 5pm, the newest copy is 15 hours old. For a portfolio you update once a month, that costs nothing. For a shop or booking site, it could mean most of a day's orders and appointments that exist in your customers' inboxes but not in your database.

As a starting point, WordPress's documentation suggests weekly backups for smaller sites with few posts, daily backups for busy ones, and always backing up the database before an upgrade.

Be equally honest about recovery time. Is getting your site back one click in a dashboard, a support ticket and a wait, or a large upload over your office broadband? The WordPress documentation notes that requesting a copy from a host's server-wide backups takes time, which is one reason it advises learning to back up your own site.

What a complete website backup contains

A typical WordPress site lives in two places. The database holds the words: the WordPress documentation says it contains every post, every comment and every link. The files hold everything else, including themes, plugins, every image and PDF you have uploaded, and configuration files such as wp-config.php. The database sits in a separate system, so downloading your site folder usually does not back it up. You need both to restore the site fully; bring back the database without the uploads and your articles return with gaps where the pictures should be.

Then consider how far back your copies go. The UK's National Cyber Security Centre (NCSC) advises keeping backups for longer than a problem might take to detect: at least a month. If a hack or a faulty update goes unnoticed for a fortnight and you keep only seven days of backups, every copy you have contains the problem.

Some of what your business relies on lives outside the site folder, so a website backup may miss it. Check yours against this list.

Keep one copy the same disaster cannot reach

At 00:47 on 10 March 2021, a fire broke out in SBG2, one of four OVHcloud datacentres on a single site in Strasbourg. That evening the company reported that no one had been injured, but that the fire had mostly destroyed SBG2 and destroyed 4 of the 12 rooms in SBG1. It later put the number of servers destroyed in SBG2 at 14,046.

Even the buildings the fire did not touch went dark. Firefighters could only work once electricity to the whole site was switched off, and the undamaged SBG3 datacentre was not operational again until 18 March. A copy stored next door to your website is little use if you cannot reach it for over a week.

Any backup kept on the same server, in the same account or in the same building as your site shares its fate: one fire, one stolen password or one mistaken deletion can take out both. The NCSC's version of the rule is 3-2-1: at least three copies, on two devices, with one offsite. It also advises keeping an offline copy separate from your network, or in a cloud service designed for the purpose, because attackers may target backups to destroy your ability to recover.

For a small website, the WordPress documentation suggests keeping copies in different places, for example one on your hosting server, one in cloud storage and one on your own computer. What matters most is that at least one cannot be reached or deleted with your hosting login.

Rehearse a restore before you need one

The NCSC puts it plainly: test your backups regularly and make sure you know how to restore files before you have to do it for real. Treat it as a fire drill. The point is to find the forgotten password or the missing folder on a quiet afternoon, not in the middle of an outage.

Name one person as the owner of the drill, so the job cannot fall between people as it did at GitLab, and repeat it after big changes such as a new theme, an online shop or a change of host. A restore that worked last year says little about the site you run today.