Thursday, March 02, 2006

Disaster Recovery

Planning for a disaster IS rocket science. I started our Disaster Recovery Plan (DRP) & Business Continuity Plan (BCP) early in 1999, soon after I came to the company. It was a perfect time. Y2K was just around the corner & all they hype had the Suits worried. That meant resources (time!) to document the entire network. As I looked at all the pieces, I wrote out an outline of what to do in case each piece went down.

Y2K was such a non-event that no one was interested in taking the next step, though. Then 9/11 came along & suddenly it was hot again, but we were busy growing & the cost of a good DRP plan was too much for our stretched resources. Memories faded & so did the DRP plan - the little bit of it that I could do.

The CSX incident in Baltimore, a water main break & a fire in an underground room have all seriously disrupted services for other companies within blocks of us, but still we're too busy to look into it. It didn't happen to us or touch our wallets.

DRP for the basic network, parts of the phone & Internet services is something I can handle. BCP is something else though. How do we put a price tag on any kind of outage? No one is really out of a job when one system or another goes down for a day or two. Efficiency takes a hit - maybe. Most of us are so overwhelmed we'd enjoy a day without the phones or Internet to actually work undisturbed.

We could lose most systems singly for a day or two without it crippling the business. It's a hassle to lose the Internet & now that we have a couple of systems with an ASP, it can really put a crimp in our style. Of course, most people can go home & work from there on those systems in a real crunch. No Internet email, but there are still phones & faxes. Most of the systems have a work around like that.

The true disaster would be if the office space got wiped out. I looked into backups for that scenario & that's where the price tag becomes too high for anyone in management to be willing to pay. Reserving space & servers is expensive. A mirrored site is out of the question for us.
So the question became moot. I made up my own plan. We have some local sites where we could stash a server or two without a problem. Some spaces where we could put a skeleton staff. I've tried to make sure that we have good backups & copies of all our documentation, vendor agreements, install CD's & licensing. I keep them stashed in a safety deposit box & with a local backup vendor. Unfortunately, time to keep all this in order is not in my schedule, so I basically keep tossing new on top of old & occasionally get months behind.

No doubt about it, if the office space disappears, it's going to be ugly, but I think the business will survive. No one will be happy & doubtless I forgot a lot of critical items, but you do the best you can with what you have. Sometimes it doesn't seem like enough, but realistically, can we really afford to do much more?

Labels: , , ,