Disaster Recovery :: Courting Disaster

• 09/01/07

##AUTHORSPLIT##<--->

Districts that are not prepared inthe event of fire, storm, earthquake,or whatever else nature may bearare asking for trouble.

AS HURRICANE CHARLEY BORE DOWN ONPort Charlotte, FL, in the summer of 2004, Charlotte CountyPublic Schools' Chris Bress made sure the small steps weretaken as well as the big ones. The district had already put itselfthrough a dry run six months earlier, an event that included thepolice and fire departments, EMS, and the local hospitals, andhad equipped its staff with new 800 MHz radios, knowing thatcell phones would fail. But Bress, Charlotte County's director oftechnology, was attempting to leave nothing to chance.

"When we knew Charley was going to hit, we told everyone to put plastic bags over their monitors and CPUs," he says. "People laughed and said the wind would blow the bags off the computers, but I told them the bags would stop water from falling on the computers when the ceilings fell in."

Score one for Bress. The ceilings did fall in, and due to his forethought, the computers were spared.

It was one of many successes for Bress and the district, whose painstaking disaster recovery planning helped get Charlotte County operational only two weeks after the hurricane struck. "Other school districts that had much less damage took much longer to open back up," Bress says.

Still, Charley took its toll, exposing miscalculations or oversights in the district's plan, in particular an underestimating of the scale of the destruction. Bress says the district anticipated a more localized disaster. "Our plan addressed losing one school, but we lost six, to the point where they would never be able to open their doors again—a total of one-third of our district in the span of one hour."

Bress estimates the cost to the school district at about $300 million, about $2 million of which was in the destruction of hardware and network infrastructure. "The amount of damage was unbelievably massive," he says.

"For a while we were so happy because we had [a plan] in writing, but the storm made us redefine the term disaster. When you think you've thought of everything, there can always be much, much more."

The ultimate lesson of Charlotte County's experience, as well as the experiences of other districts that can share their own war stories, is plain: Be prepared. When that storm has you in its sights is not the time to be fumbling for a disaster recovery plan. A strategy better be in place and account for every foreseeable outcome. An inadequate or altogether nonexistent plan can have dire results. Fire, power outage, severe weather—all can bring down even the best of networks in an instant.

Nederland Independent School District in Texas, which didn't have a formal disaster recovery plan before Hurricane Rita hit in September 2005, made it a priority afterward.

Rita inflicted $10 million in damage on the school district, including buildings and infrastructure, although Cindy Laird, Nederland's director of instructional technology, points out that the majority of the damage was structural. "We had secured our equipment, so if we lost any equipment, it was actually due to roof leaks," she says.

"We knew things that needed to be done because we had been hit by tropical storms before, but when it actually came to Hurricane Rita, I learned quite a bit." One of the things Laird learned was the importance of having mail servers off-site.

Gather software and hardware informationin one binder for easy reference."You have one communication folder, so you're not scrounging for model numbersand phone numbers."

—Cindy Laird, Nederland Independent School District

"E-mail was the main source of communication—we were able to communicate throughout [the event]," she says. "[With the mail servers off-site] I still had access and I could get word and news out to the community—the superintendents could put letters or messages on the site daily to let people know what was going on."

That's a lesson learned echoed by every educator who has been through a disaster: Communication is critical during times of crisis, so keep some avenue open, be it e-mail, phones, or internet. To that end, some school districts are looking at or adopting strategies such as off-site storage and management of communication systems, including e-mail and telecom.

Bress credits Charlotte County's wireless broadband network with enabling the district to stay in touch with parents, students, teachers, and administrators after Charley hit. The district installed the wireless network about a year earlier, mostly because some of the older schools did not have the infrastructure to support traditional wired systems, Bress says. It was a bonus that the setup got communications up and running in just a matter of days after the storm blew through.

"After the first few days," Bress says, "we were able to get a big generator attached to one of our schools that could provide power to radios, and we had internet activity back to that school. That was critical because it would take six weeks to get phone communications to that part of town. We could allow people to come in and send out e-mails and let people know they were okay.

"In a situation like that, it's bad enough that someone's world has been turned upside down, but if you can give them something that's somewhat normal, it gives them a better, safer feeling."

Setting Priorities

The catastrophes that attract the most news coverage are of the natural kind— earthquakes, hurricanes, tornadoes. But those aren't most commonly crippling to schools. According to a white paper by Washington-based education solutions provider ESP Solutions Group, loss of electrical power is the most frequent cause of a disaster in the K-12 space.

"When we put together our plan, we had to define what a disaster is for inbuilding, out-of-building, and regional disasters," says Tom Petry, director of technology at Collier County Schools in Naples, FL. "What else can happen besides a hurricane? So we included local power outages for extended periods of time, fires, etc. We absolutely did not want to have a disaster recovery plan that takes 24 hours to implement, and that was the starting point we worked from."

Indeed, disaster recovery plans should deal with myriad scenarios, from a malfunctioning sprinkler system to a school lockdown. IT administrators say that the top three considerations in any plan, for any situation, should be:

• communication (phone and/or e-mail)
• line-of-business applications (payroll, human resources, etc.)
• student information/administration

Petry says prioritization is critical in keeping operations running smoothly. "You can't have everything back up and running in eight hours, so you have to prioritize what you need in order to function," he explains. "For us, financial apps, e-mail, and VoIP fit into our eight-hour window. Instructional apps,however, don't fit." He adds that everysoftware program seems essential, "butyou have to figure out what is actuallymission-critical."

What's needed is a kind of tech-support triage, because once the storm strikes, Bress says, IT gets pulled in every direction: "No one is as popular as tech support is when things go down."

In particular, Bress says, he and his staff "didn't take into consideration how much other departments would need us." He describes how the magnitude of the destruction that Charley wrought forced the district to change its priorities on the fly. Restoring service to the payroll department became an immediate priority, so people could get paid. "As long as their software and hardware is working, we normally never hear from payroll with respect to paychecks," Bress says. "We had to disassemble all of their equipment and move it north until we could find a secure location and steady power."

The transportation division also was moved up to the front of the line. Transportation plans had to be changed for more than 17,000 students, many of whom had never been on a school bus before because they attended neighborhood schools. "Our transporation department could not even look up the home addresses of students, let alone change their schedules," Bress says, "because their offices were located right in the middle of the destruction zone. We had to disassemble all of their equipment as well and move them to an adjunct transportation facility on the other side of town. They were anxious to get to work, but they couldn't start until we got them hooked back up."

An Enterprise Model

As part of their planning for more than the obvious disasters, some school districts are lifting their approach from the business realm, building and maintaining healthy, strong, and flexible disaster recovery and business continuity plans.

"Since Katrina, there has been more heightened priority in having a disaster recovery plan," says Eric Schott, director of product management at data storage systems vendor EqualLogic in Nashua, NH. In education, Schott explains, similar to business, "an IT person is always matching the [district's] services needs against the threat of the service and business objectives of an organization. In that respect, an IT person in K-12 is no different from an IT person in a smallto- medium business."

According to research and consulting firm Frost & Sullivan, enterprise/ business continuity and disaster recovery spending totaled $15.1 billion in 2006 and is estimated to reach $23.3 billion in 2012. While no figures are available for disaster recovery spending in the K-12 space, anecdotal evidence suggests school districts are upgrading their IT infrastructure with an eye toward minimizing— or even averting—any downtime in the event of a disaster.

"When we developed our disaster recovery plan, we took some cues from the business world," says Keith Price, chief technology officer for Hoover City Schools in Alabama, which fortunately hasn't had to implement its disaster recovery plan yet, despite being near an area known as Tornado Alley. "We consulted with our technology partners to see what they had done with other customers [in different verticals]," Price says. "We wanted to look at their best practices and figure out what worked best for them. I feel like we've had similar results."

Decentralize your data servers."The problem with centralizing your data is, if you get hit, you lose everything."

—Tom Petry, Collier County Schools

"With the way technology has moved, it is becoming more cost-effective for school districts to do what enterprise is doing; the only constraint is the budget," says Renaye Thornborrow, director of marketing at Trillion, a broadband managed service provider for the K-12 market based in Austin, TX. "They are all looking for ways to reduce costs while still being able to provide effective technology."

Granted, school districts and enterprise companies draw up and adhere to different sets of priorities in their disaster recovery plans. But the underlying goals are the same: restore operations as soon as possible, with minimal, if any, loss of data.

In addition, both business and education have to be mindful of the potential leakage of confidential and/or personal data—from unsecured networks after the event, theft of backup tapes, or other similar situations—that could put them in a compromising situation.

Bress says a school district's disaster recovery plan will be different from, say, a financial institution's "because we don't have to carry credit card or other data like that," he explains, noting, still, that the two organizations have more in common than not. "We do have student data and health records. When you're comparing the two, there might be 70 to 80 percent similarity and then 20 percent specific to what you do."

In fact, Bress says the single most important piece of technology in his district's disaster recovery plan was the data backups: "Hardware can be replaced, data can't."

Petry had the foresight to change Collier County's network setup before the district was struck by Hurricane Wilma in 2005. "We had spent a lot of money to decentralize our data servers," he says. "The problem with centralizing your data is, if you get hit, you lose everything." Fortunately, Wilma did little damage to the school district—one wall of a school was destroyed and the landscaping was ruined—but nothing else of any significance.

The unfortunate reality is that all the calculation, anticipation, and planning can't entirely prepare a school district for the impact of a natural disaster, or protect it from widespread destruction. Some things simply can't be predicted, or in the worst-case scenario, such as Katrina, even imagined.

"The only thing that I thought we were unprepared for was fate itself," Bress says. "God forbid we ever have to face something of that magnitude again, but if we did, I am sure of only one thing: It will not be exactly the same. Priorities would once again be shifted to meet the unique nature of what was happening."

"Until you actually experience it, you don't know what you need," says Nederland ISD's Laird. "It was unfortunate that we had to go through it, but it was a great learning experience."

-Charlene O'Hanlon is a freelance writerbased in New York.

This article originally appeared in the 09/01/2007 issue of THE Journal.