Trussville City Schools Regains Control of Its Network

• By Dian Schaffhauser
• 09/23/08

##AUTHORSPLIT##<--->

If you were designing a new school district network, what forms of security would you implement? That was the challenge faced by Shawn Nutting, director of technology for Trussville City Schools in Alabama. On July 1 three years ago, Trussville became an independent school district when it broke off from the considerably larger Jefferson County School System in Birmingham, AL.

Armed with a supportive school board and what he calls a "realistic budget," Nutting and his team of seven, including five computer technicians, had 30 days to build a network that would support five schools, 5,000 users, and 2,000 computers and one that over 10 years could grow to support a one-to-one program.

In the network closets, he ran what he considered top-notch equipment: Cisco switches and routers, firewall, and VPN concentrator and a deep-packet inspection appliance from Top Layer, a wireless system that external users would never know existed.

On the user security front, the district chose Symantec for anti-virus. But Nutting was also convinced that whitelisting--specifying in advance what software could be installed and run by which users--was the way to keep machines clean.

"When I came here three years ago, before we split off from Jefferson, no computer in the computer lab could get to the Internet," said Nutting. "You'd click on [Internet Explorer], and 50 million casino gaming pop-ups would fire up."

The teachers would prefer having free reign over what they can run on their computers, said Nutting. "But as a network professional, it's obvious to see the advantage of application whitelisting. We have 20,000 executables on our network now. We can't grow to 100,000. There's no way five people can handle that. We need to get it down to where it's manageable and only run software that we really need."

As a Microsoft shop, Nutting decided to use Windows Server Group Policy to manage applications on user machines. But that decision had its flaws. "Group Policy does one heck of a job of keeping you from installing Windows software," says Nutting. What it didn't do so well was keep script kiddies in his high school from doing ping sweeps or port scans. "We quickly realized Group Policy wasn't going to cut it for what we needed."

As the next step, the district chose Sanctuary from SecureWave, a company that has since been acquired by PatchLink and renamed Lumension Security.

But problems surfaced whenever the team pushed out a Windows update under Sanctuary. Suddenly, for certain users, programs previously approved weren't approved anymore, and their computers would stop working.

As Network Administrator Cameron Sewell explained, "Even though we did our best and followed directions to get every single piece of software approved through Sanctuary, there would still be certain things that wouldn't run." In order to get around these problems and enable users to have access to their computers again, IT actually had to introduce a security hole. "We had to create what's called path rules in the application that would leave an entire directory open. Anything run from that directory would be approved."

That meant that if a high school student at the time were smart enough to get around Group Policy and find the Adobe directory, for example, he or she could stick anything in that directory and get it to run from there.

Also, a limitation of the software was that it worked at the user level, not at the computer level. That meant IT couldn't enable an application to run on a given computer no matter who was using it, such as in a computer lab.

Those aspects of the solution were frustrating enough, said Nutting. But what finally made the district move away from its previous security solution was the number of bluescreens it was experiencing. Sewell estimated that during the height of the problems, whenever IT rolled out an operating system patch or update, they'd get 20 to 40 computers in a week that had stopped working.

"When a teacher loses a computer, one of these poor guys has to go and literally spend three and a half hours reinstalling software for that teacher," said Nutting. "It became a security gamble that the best thing to do was to stop sending out Windows updates."

The district found its technical support for the product coming from Luxembourg. "It wasn't instantaneous help because we were in a different time zone," said Nutting.

Committed to the concept of whitelisting, Nutting had Sewell evaluated alternatives, including offerings from Symantec and Bit9. They were hoping to find a solution that offered more automatic methods for approval, as well as something that would work at both the user and computer level. After running demos and trying out the software on the network, the district chose Bit9's Parity because it fulfilled the majority of requirements.

The district included Parity as the image rolled out to all computers to prepare for the fall 2008 semester. The company sent a couple of technical people to the district to install the software, get the base image set up, and troubleshoot quirks on the network.

A dedicated server running in a virtual machine maintains the management functionality. It communicates with a SQL database containing the list of rules and approved applications. Every computer running on the network has a client application that communicates with the server to get updated.

"You could deploy a Bit9 client to everything, but that remote deployment approves everything on the computer at that time," said Nutting. "We took the cautious step of going back, cleaning everything up, and making sure it was the way we wanted it to be before we pushed the Bit9 client out." That ensured that viruses wouldn't slip through and that Sanctuary was removed from the computer.

IT set up a logon policy that if a computer has neither Sanctuary nor Parity on it, the user couldn't log in. Nutting said there are still about 150 computers in the district that are still on Sanctuary that need to be maintained.

Now any software on the approved list for a given computer or user can be installed by that user. Nobody has to ask first. "We don't lock them out from installing their own software or printer," said Sewell. "We just have to control what they have to install."

If there's a program a teacher wants that isn't on the master list, he or she must put in a work order. One of the computer technicians approves the software and makes it available on the network.

Starting in December and January, the district will have a new process in place. There will be two periods during the school year in which IT will install new software. And it will be approved first by a committee consisting of teachers from the school where the request is originating from, as well as the district curriculum director. "I'm not a teacher," said Nutting. "I don't need to be saying whether Jumpstart [World] Kindergarten is a good program or not. I don't have enough experience to approve that."

Application whitelisting requires a lot of management, said Nutting. "But I still emphatically believe that the advantage for us as far as having a clean, efficient network without a lot of malware and junk is very worth the hassle on the front end. In a school system, it adds pressure because teachers complain, and in traditional school system rules, anyone who complains gets what they want. But no amount of complaining will make us relax our security settings."