Once Again, The Day is Saved!

(Actually, as of this writing, I'm not entirely positive that the Day is Saved... but it's pretty likely... check back for updates.)

Have I mentioned what it is Kevin does for a living? He babysits a bunch of servers that hold data from pharmacutical companies - which reps visited which doctors and gave out samples, etc etc. This data is held in a database, then backed up and stored off-site. Standard data-storage procedures, really...

Last week, one of the big, big drug-manufacturing companies - in fact, the biggest, and Kevin's company's biggest client (they account for more than 40% of Kevin's company's business) - requested some year-end backups from 2003 that they needed for a lawsuit they're currently involved in.

So.... important data. And a year-end back up is - surprisingly enough - only done once a year.

The request comes in for the data, the shift on duty (not Kevin's shift) checks the company's log listing for what data they have on the off-site. The year-end back up was missing from the list.

Not. Good.

Except that Kevin knows that many of the other employees aren't as vigilent about maintaining those logs as they should be. So he called the off-site storage company and had them send him a list of what they had. There was discrepancy... the year-end backup was there, it just wasn't listed on the DCS's logs. (Data Center Services)

In any case, Day is Saved part one. Kevin finds the missing tapes and recalls them. The tapes arrive on site. Kevin, as is proper, goes to the DBA and turns over the tapes, which are then put in the server where this sort of recovery takes place.

The DBA says, "I'll write-protect these, ok? And leave them here until I'm done, ok?"

Kevin says, "Ok."

Kevin... goes home from work.

Less than an hour later, we're at Jess's playgroup and his phone rings. He talks for a while, then looks somewhat pale.

After he gets off the phone, Kevin tells me that one of the night-shift operators accidentally deleted the year-end backups...

The DBA in question hadn't saved the recovered data that he was working with...

He also - the DBA, I mean - hadn't write-protected the tapes.

However, it is possible to recover data from tapes like this - an erase doesn't actually erase the data, it just erases the table of contents for the data. (It's like taking a 1,000 page doccument and just tearing off the index... the information is still there, you just can't find any of it easily.)

Kevin immediately upon arriving home, started researching software solutions to recovering the data. Amazingly enough, this doesn't cost very much... so, barring more unexpected fuck-ups, the Day is Once Again... Saved.

Posted by tisfan at February 09, 2005 10:18 AM