Backup your backups

Posted: November 20, 2010 in backup, computers, hardware
Tags: , , , , ,

There is an old saying “there are those that have already suffered hard-drive failures and those that are about to“.  This was ringing all too true for us on both counts recently.

I’m always pretty good at ensuring all the important info we have personally and as a company is backed up, or at least I thought I was.  As it turns out, one of the Western Digital RAID NAS (Redundant Array of Independent Disks & Networked Attached Storage)  we use failed.  It did give some warning signs, but they were chalked up to overuse and taxing of the NAS during large backups.  Add to that the fact the NAS was having issues I was not made aware of until after the fact and you have a recipe for disaster.  Little things like having to restart the NAS every few days should have been a clue something was up, but when more than one person is doing it and not telling everyone about it, the problem doesn’t really get noticed.

Thankfully though, as mentioned above the NAS was set up as a RAID 1, which meant the data is duplicated on each of the two drives inside the enclosure.  In case one fails, you should still have the other to retrieve from.  RAIDs are funny things though.  If the RAID controller fails, then you can’t access the info on the drives because of the way it is stored.

Removing and hooking up the SATA drive to a USB2 adapter does you no good.  All it will tell you is the size of the drive and whether its healthy and how much info is stored on it, but not actually let you see any of the data or access it easily.  It was at this point I started looking around to see what was on that drive we needed that wasn’t also stored elsewhere.

Most of the info on the NAS was backed up onto other externals so it really wasn’t a big deal that we couldn’t access it.  A little moving around of the backup info onto another NAS and we hardly noticed the problem with one exception.  All of the digital photos.  We use them to remind us of what we saw on site, etc, so having them not available was a bit of a pain.  We had an old backup but it was many months old which can amount to many gig of missing images.  We sent the NAS off at a local data recovery company and hoped for the best.  A couple days later we got a complete recovery back on a loner external drive.  After a few hours of data comparison and merging of files we are back to working at full speed again.  The first thing I did when we had all the data back in place was a complete backup of those files in the hopes this never happens again.  Live and learn.

As for the faulty Western Digital NAS, it was still under warranty and I took advantage of the express RMA agreement which means they send you the replacement first and after receiving it you return the faulty one within a certain time period or else they charge your credit card for the replacement.  The new one is a newer model, and seems better thought out with respect to what the light on the front is telling you.  The old double blue ring style we had was confusing as to what it was indicating.  The new one is simpler.  A single lit line that shows the amount full and also indicates whether it is being accessed.

What happened to us could have been much worse and we were lucky, but regardless the lesson is the same and I hope you all take it to heart.  Backup your backups.  There is no such thing as being too prepared, or backing up too often, or to too many places.  It may seem pessimistic but you’ll never be disappointed if you expect the worst and plan for it when it comes to hard-drive failure.

tcg.

Advertisements

Please Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s