Fail Safe: Data Security at Hendrick Hudson School District

• By Dian Schaffhauser
• 06/23/09

When a district has a nuclear power plant not two miles from one of its elementary schools, concerns about disaster recovery are bound to surface in technology discussions. Yet it took Mathew Swerdloff, director of technology for Hendrick Hudson School District, three tries and three years before he found a system he could trust for maintaining and archiving data.

The district, with 500 employees, has three elementary schools, one middle school, and one high school, all located about 35 miles north of New York. Buchanan-Verplanck Elementary School in Buchanan, NY is situated 1.6 miles from Indian Point Energy Center, a three-unit nuclear power plant also located in Buchanan, along the Hudson River.

Tech Makeover
When Swerdloff arrived in the 2,800-student district about five years ago, the infrastructure was in bad shape, he said. "There was not a regular budget for technology, there was no technology director. Computers were seven or eight years old. Servers were as old or older than that. There was no high speed [wide area network]. We've had to redo everything."

He and his IT team--two network specialists and one audio visual specialist--set about replacing all of the district's servers and PCs and putting the hardware into a five-year replacement cycle. The servers were consolidated to a central location. More recently, servers were migrated from Novell to Windows Server 2003. E-mail has been moved off of Novell GroupWise and onto Microsoft Exchange. Plus, the IT group has deployed a 3 TB storage area network (SAN).

The backup process for critical data consisted of running tapes in an eight-day rotation cycle. Since servers were scattered, the tapes would be shuffled among buildings in the district. But that approach provided "no peace of mind," Swerdloff observed. "We had numerous instances--and I've had it in other districts too--where servers blew up. And the tapes are just not reliable. We'd have to rebuild a server and get certain things working just right. But I never felt confident that the tape data was going to be all there and accurate and easy to get."

So two years ago Swerdloff began looking into online backup, where the data would be hosted offsite by a vendor. He didn't do anything as formal as a request for proposal. He simply sent off e-mail to various vendors and ended up trying three or four of their products on one of the servers.

From his limited testing, Swerdloff chose to contract with intronis, which provides online backup services and restoration. For the next several months, he and his team struggled to configure the backup process to work with the service--with mixed results. "It was very easy to use," he said. "But we had trouble with the plugins for Exchange and SQL Server. We never got Exchange working." When the company suggested a quirky workaround, he said, "That was the final straw for me."

His mistake, he realized, was in not testing the service on every server that would be backed up. "I didn't do that, so I didn't get an accurate picture for how it was going to work."

However, with the new service in place, the district stopped doing the tape backup. But as soon as the difficulties surfaced, Swerdloff began doing manual file copies of critical data from one server to another. "That wasn't ideal," he pointed out.

About four months into the contract for the new service, Swerdloff reconsidered his options. This time he put together a formal RFP, "because I knew a lot more about what we needed." As part of that process, he specified that the district wanted to test a full-working version that had no limits for 30 days.

He sent the RFP to a "bunch" of vendors and received five or six responses. After looking those over, he selected three vendors to supply software for evaluation.

Swerdloff installed each product for two weeks on almost every server in the district. "Every server has its own little unique configuration," he said. "We have two servers running SQL Server. We have a couple of proprietary databases that I wanted to get backed up. Exchange is another big one. We have another one called [Total Traffic Control from Lightspeed Systems], our Web filtering software. That's a huge amount of data--a 70 GB SQL database. So I really wanted to make sure it was going to work. That was the only way to make sure."

"I did the installation for two weeks, and then uninstalled it," he recalled. "I did the second one for two weeks and uninstalled it. And the third one was [EVault from] i365, which I just left on, because it was the best in my opinion."

He signed a three-year contract for the new service, and, for a while, the district ran both services in parallel, until the contract with intronis ran out. To reduce the burden of double costs, he said, "since we were paying for capacity, we cancelled all the jobs on the first service, so we were just paying a nominal fee."

What's to Like (and Not Like) about i365 EVault
i365 was formed in 2008 by Seagate, which rebranded three of its companies, EVault, MetaLINCS, and Seagate Recovery Services, which had been operating as a division of Seagate Services.

EVault is available in four forms: as software that an organization can use on its own servers, as an appliance that's installed in the client environment, as a managed service in which i365 handles the backup work for the customer, and as a software-as-a-service (SaaS) offering that backs up the data to servers hosted by i365 but under the control of the client. Hendrick Hudson uses the last solution. As Swerdloff explained, each server in the district runs agent software, and he and one of the network techs each have a console on their computers to manage the jobs. The data is stored offsite to an i365's data center and replicated to a second data center.

After an initial backup, EVault copies only changed data blocks, which translates to speedy backups--about 70 GB in about 15 minutes, Swerdloff estimated. When a file needs to be restored, the software recreates the entire file on the fly and returns it to the user. "Four or five times this year we've had instances where somebody lost something, and we were able to retrieve it within 10 minutes. It's just a matter of a few clicks," he said. "I didn't have to go find the tape then take 15 minutes to rewind the tape, then discover it's not the right tape and have to go back down to track down the right tape. I can do this right from my desk. It's like using Windows Explorer. I browse to the file and say, 'Restore.'" Access to data can be done from any location with Web access.

There's much to like about how the service works. The management is "simple but robust," and it allows Swerdloff to do "some pretty sophisticated filtering." That comes in handy in specifying what types of files get backed up and what folders they're backed up to. "It's easy to say, 'Back up everything,' but that just costs too much," he explained. As the amount of data backed up grows, so does the fee. The budget for storage started out at $20,000, but it has since risen to $30,000. That's the price the district pays to back up about 400 GB in compressed form annually from its 11 servers.

Also, the district paid a one-time charge for the specific plug-ins it needed to back up SQL Server and Exchange data. Paying that extra charge rankled Swerdloff. "I did my best to negotiate that, but I didn't like it," he added. Overall, cost is the biggest negative in using i365, he has found. "Once you get past that, I see no negatives."

Part of the funding for the online service came as a result of district operations audits done internally and by the state, which coincided nicely with Swerdloff's desire to move away from the tape backup burden. "In both cases they said they needed to see that we had a disaster recovery plan in place and a way to ensure continuity of operations in the wake of a disaster," explained Swerdloff. A portion of the expense was re-allocated from the budget for the tape backup system, which included the cost of buying 80 tapes a year at $80 each. The district dedicated additional IT budget to get the new service implemented.

To address the growing expense, IT has imposed storage quotas on all users and placed restrictions on the size of e-mail data stores for faculty and staff. "Now on a case by case basis we're giving people more space," added Swerdloff. He also has created custom configurations that the service follows automatically for different types of data. For example, he doesn't bother backing up student network shares, photos, or videos.

Data Safety and Peace of Mind
Next up, Swerdloff and his team will be converting their stand-alone servers into virtual servers running VMware software and implementing a second SAN for mirroring to attain multiple points of redundancy.

"There's always talk about what's mission critical. A few years ago I would have thought student and teacher files were not mission-critical. Payroll was," Swerdloff said. "But now I know, if a school closes and has to be re-opened in a temporary location, those teachers will need all their files."

In reference to the nuclear power plant down the road, he concludes, "You don't want to think about worse possible scenarios; but if we had an incident, this district would be closed for months. And no one could get in here to get the data--I'm not coming in to get a server. We have to have a way to be able to recreate everything."

Now, he observed, "All of our data is offsite in two places. It's total peace of mind--the data is safe."