hard drive, disk, hardware, data, hard, technology, computer, information, storage, drive, hdd, pc, memory, disc, device, computing, gray computer, gray technology, gray laptop, gray data, gray information, gray memory, hard drive, hard drive, hard drive, hard drive, hard drive, hdd, hdd, hdd. Why does a backup keep reporting success while it quietly fails?
Photo by bohed on Pixabay

Maintenance

Part of Backup and recovery: a grounded overview

Why does a backup keep reporting success while it quietly fails?

Ten ways a backup arrangement fails while reporting success, why the status stayed green each time, and the routine check that would have caught it months earlier.

Backup failures are quiet. Almost none of them announce themselves; they are discovered on the morning somebody needs a file, which is the worst possible moment to learn anything. That quietness is the actual problem to solve, and it is why this page is organized around what the arrangement was reporting at the time rather than around error messages.

Each case below gives the failure, why nothing looked wrong, the check that finds it early, and the fix. Where a fix is tempting and wrong, that is said too.

What to take away

  • A green status reports that a job ran, not that a copy is readable.
  • Every one of these is found by restoring something real on a schedule.
  • Fix the reporting first. An arrangement that cannot tell you it is broken will break again silently.

The job succeeded and the archive will not open

Why nothing looked wrong: verification compared a size or a checksum written at the same time as the file, so a corrupt copy verified against itself.

Verification vs. real restore

What was checked

File size
File appeared
Checksum at write time
Copy matches itself
Quarterly restore of contents
File opens

What it proves

File size
Not readable
Checksum at write time
Corruption hidden
Quarterly restore of contents
Corruption caught

Found early by: a quarterly restore that opens the contents rather than confirming a file appeared.

Fix: turn on whatever content-level verification the product offers, and add a restore of a real file to the calendar. Where the product cannot read content back, that limitation is now a known property of your arrangement and belongs on the written page.

The scope shrank and nobody was told

Why nothing looked wrong: the job covered what it was told to cover. A share added in spring was never added to it.

Catch scope drift quarterly

  • Compare report's copied list to system inventory
  • Select by system, not folder path
  • Turn on skipped-item reporting
  • Check the first report

Found early by: comparing the report's list of what was copied against the system list from your own inventory, once a quarter.

Fix: select by system rather than by folder path wherever the software allows it, and turn on skipped-item reporting. Then check the first report, because there is always something.

Notifications have been failing into an unread mailbox

Why nothing looked wrong: it was reporting loudly, to nobody. This is the most common backup failure in small businesses by a wide margin.

Prove a human reacts

  1. Rename a source folder deliberately
  2. Wait one week for a human reaction
  3. Send failures to a route the person reads
  4. Name that person
  5. Repeat the deliberate break annually

Found early by: deliberately breaking something, such as renaming a source folder, and seeing whether a human reacts within a week.

Fix: send failures to a route the person actually reads, name that person, and repeat the deliberate break annually. A weekly summary short enough to read beats a daily one nobody opens.

Refuse: adding a second dashboard.

The backup window overruns into the working day

Why nothing looked wrong: the job completed, just later each week as the data grew.

Read duration, not status

  • Job statusgreen
  • Durationlater each week
  • Datagrowing
  • Windowfixed

Found early by: reading the duration column rather than the status column.

Fix: move the largest system to its own schedule, reduce how often the least important data is copied, or change the copy type so that full copies happen less often. Growing data with a fixed window is a capacity problem, not a fault.

A restore is far slower than anyone expected

Why nothing looked wrong: nothing was wrong. The number had simply never been measured.

Speed vs. safety trade

Copy

Local copy
Speed
Remote copy
Safety
Measured restore time
Decides per system

What it gives

Local copy
Fast restore
Remote copy
Off-site protection
Measured restore time
Not assumed

Found early by: restoring a whole share once and writing the time down.

Fix: if the measured time is longer than the business can bear, the answer is a local copy for speed alongside the remote one for safety. That trade is the reason the two numbers in backup and recovery are decided per system rather than once.

Refuse: assuming the number will be better under pressure.

The key cannot be found

Why nothing looked wrong: the key worked every time it was used, from inside the environment where it lived.

Key custody checklist

  • Write down what each key opens
  • Store a copy outside every protected environment
  • Have a second person retrieve it unaided
  • Put key custody on the departure checklist

Found early by: having a second person retrieve the key, from outside, without help.

Fix: write down what each key opens, store a copy outside every environment it protects, and put key custody on the departure checklist alongside accounts, in the way identity and access treats any other privileged credential.

An administrator deleted or shortened something

Why nothing looked wrong: the change was permitted, so no alarm existed to sound.

Unexplained retention change

Was the retention or job change explained?

Yes

Permitted change, alert on the setting

No

Start incident response sequence

Found early by: an alert on retention changes and job deletions, tested once.

Fix: move at least one copy onto storage where retention cannot be shortened by an account holder, and alert on the setting regardless. Where this happened without explanation, stop treating it as a fault and start the sequence in incident response, because it may be somebody else's change.

The database restores and the application refuses it

Why nothing looked wrong: the files were copied faithfully, mid-write.

Found early by: restoring the database into a scratch environment and opening it with the application, not with a file browser.

Fix: ask the software supplier what a consistent copy requires. Usually it is an application-aware capture, a database export, or a copy taken while the application is closed. The backup vendor cannot answer this question for somebody else's software.

Everything is backed up except what lives online

Why nothing looked wrong: the arrangement was built around machines, and the mailboxes and shared documents live somewhere else.

Found early by: listing the online services the business uses and asking, for each, what independent copy exists.

Fix: point the arrangement at the platform, or export what cannot be recreated on a schedule. What a provider retains for you is not a backup, for the reasons set out in cloud security.

The restore worked and the problem came back

Why nothing looked wrong: the restore was successful. The copy simply predated nothing.

Found early by: retention that reaches back further than the time it typically takes to notice a problem, which is longer than most people set.

Fix: extend the long tail of weekly and monthly points, and before restoring, choose a copy from before the earliest evidence rather than the most recent one. Change credentials before the restore, not after.

Refuse: restoring into the same environment, with the same credentials, because the business is impatient.

The pattern behind all ten

HabitWhat it catches
Restore one real file quarterly and open itCorruption, key problems, console unfamiliarity
Restore one whole system annually and time itDuration, consistency, hardware assumptions
Read the duration column, not the status columnA backup window quietly overrunning
Compare the report's coverage against your system listScope drift
Break something deliberately once a yearNotifications going nowhere
Alert on retention and job changesDeliberate interference, and mistakes

Six habits take perhaps three hours a year. Every case on this page is caught by one of them.

That's the argument the United States Cybersecurity and Infrastructure Security Agency's Cyber Essentials makes for a small number of actions done regularly, not a larger number done once.

When it is not a fault

When backups fail, retention shrinks, or jobs vanish with no change to explain it, do not treat this as a troubleshooting exercise. Unexplained interference with recovery capability is a possible incident. Act on it accordingly.

The United States National Institute of Standards and Technology's computer security incident handling guide treats this as an early indicator, not a maintenance issue.

Common questions

The last three months of backups are unreadable. Can anything be recovered?

Sometimes, from an older independent copy, which is the argument for periodic full copies and a long tail. Stop writing new copies over anything before investigating, and get help before experimenting on the only remaining data.

We have no failures at all. Is that suspicious?

It is worth confirming rather than enjoying. Break something deliberately and see whether the arrangement notices. Silence you have not tested is indistinguishable from silence that is broken.

How much of this can a managed provider handle?

The operation, all of it. The proof, none of it. Ask for restore test dates and durations in writing, and keep the written page yourself, as the routine in security foundations recommends for anything a business depends on but does not operate.

Our arrangement fails one night in ten. Is that acceptable?

Only if you know why and the missed night is covered by the next one. Intermittent failure with no known cause is a fault waiting for the worst possible night, and it is worth an hour with whoever supports the product.

More in Maintenance

Latest from Value Desk