The Benefits of an IT Backup Plan

Technology managers always face tough decisions when developing disaster recovery and continuity of operations plans. Sometimes, even when an IT team makes as responsible a financial decision as possible, an unexpected event can cast doubt on that decision.

Such was the case at the University of Virginia, where an intense storm in June buffeted the Charlottesville area with winds exceeding 75 miles per hour. In just a matter of minutes, downed trees and power lines filled the roads.

The storm hit late on a Thursday afternoon; by Friday morning a backup generator and uninterruptible power supply at the university had failed, causing critical e-mail and web servers to go down and stay down for 11 hours.

Michael R. McPherson, associate vice president and deputy CIO, says the irony of the outage was that the IT staff had budgeted for a new redundant data center that would have easily weathered such a storm. But that facility is not slated to come online until next spring.

The IT team had weighed buying a new backup generator. But calculating risk against expense favored the decision to forgo that purchase and funnel the funds into the new data center project. One approach took the long view; the other, the short.

“What this shows is that real events can expose the flaws in your plans,” ­McPherson says. “It reminds you of the trade-offs you make while developing these kinds of disaster recovery plans. It just made more sense to us at the time to budget for a new data center over a couple of budget cycles. The new center will give us a huge level of redundancy.”

Real-Life Decisions

This story's poignancy derives from the fact that it comes from real life. In these tight budget times, every IT staff must weigh similar decisions. College and university CIOs likely can identify with the Charlottesville experience because it's more in line with the kind of decisions and events most IT managers at college campuses face.

High-profile disasters such as shootings grab headlines and certainly require that colleges bolster emergency response systems. But they don't represent the day-to-day COOP realities at higher education institutions.

Nothing can replace sitting around a conference table evaluating the pros and cons of all the nuts-and-bolts decisions that IT staffs must make. Do we build a new data center a few miles from campus that for several years to come could keep us running through natural disasters? Or do we outsource data services to a third party and maybe have the money for a new generator, too? Will virtualization or cloud computing help us with disaster recovery? Is it enough to outsource e-mail so people can receive messages during a power outage?

The truth is that nobody really knows the absolute best way to make these difficult decisions, says Joel Hartman, vice provost and CIO at the University of Central Florida. “The reality is that you can't be sure of what you've never experienced.” You must identify the risks of each approach, calculate the costs and then prioritize your options. Sometimes unexpected events may prove you wrong.

That's why Hartman, who works in a state with a six-month hurricane season, says he'll never declare his continuity plan foolproof. You can't be ready for everything, Hartman reasons, but you can do your best to be ready for anything. Because, as UVA's McPherson aptly puts it, “You can't control when the disaster happens.”