How Software-Defined Networking Raised One District's IT Response

A continuing expansion of digital initiatives at Carmel Unified forced the school system to find a more effective way to stay on top of IT demands. SDN is delivering the goods.

• By Dian Schaffhauser
• 05/23/19

Before Carmel Unified School District began its software defined networking journey, Network Administrator Rob Perry spent a lot of time putting out fires. He would get constant complaints of outages. Students would open their Chromebooks in the classroom, and, depending on the site, "half of them" wouldn't be able to connect to the network. Or in kindergarten rooms at one of the elementaries, any time more than one person tried to get on the network, operations "would grind to a halt."

Perry has been in network management for more than two dozen years, and when he joined the California district in 2016 it didn't take him long to figure out that the state of the network was "very patchwork." Some parts were in "really good shape" and others "looked like they hadn't had much attention."

Some of that was budgetary. There were only two central IT people besides him: the chief technology officer and a data specialist. Another three site technical support assistants were spread across eight sites. And there was far more work to be done than anyone had time to handle. At the same time, the schools were deep into their digital transformations. Grades 3 through 12 were already 1-to-1, and other network demands were really beginning to heat up.

An infusion of new IT hires and reorganizing of the existing tech team lessened the pressures on staff (which is now up to 11 people). Perry was able to employ a systems specialist to take over some of his duties, including server and phone system management. That freed him up to start focusing on the infrastructure.

Picking the Low-hanging Fruit

As he began digging in, he found a lot of "really old equipment" and "aged" wiring. For example, when he started running tests at that kindergarten, he found users were sharing a 50 kilobit/second link, in spite of the fact that they were on fiber. Not only was the connection spliced with different types of fiber, but the switch that it was going to wasn't configured properly.

So, Perry's first major effort was to get the wiring upgraded. The district paid for the systems specialist to get certified as a fiber installer, and he went into the schools and pulled "many hundreds of feet of new fiber" as part of the project.

Next up, getting the switches upgraded and putting schools in the position of being able to exploit the 10-gigabit backbone that was already in place.

Those changes in themselves made a difference "of night and day" for users, said Perry.

Suddenly, it was no big deal when a rival team was playing at the high school stadium, drawing record crowds, and everybody wanted to post pictures, videos and comments to Facebook or Twitter; or when people making interactive presentations in the high school theater would ask the audience to pull out their phones and get on the guest network to fill out a Google form or give online responses. Teachers didn't have to worry about how well their new virtual reality labs, robotics classes or makerspaces would hold up under the network demands.

Then the work was helped along by the hiring of a new superintendent, Barb Dill-Varga, in mid-2017, who put tech at the top of the to-do list. In late 2017 Perry began work on a proposal that Dill-Varga called "Project 2020," a two-and-a-half-year plan to update the entire network infrastructure for "a modern educational environment." But the superintendent informed him that his timeframe was too long. She wanted it done faster. "She was a huge source of support in making sure we had the funding we needed and accelerating our timeline," Perry noted.

The Allure of Automation

But upgrading the physical network was only part of the job. The next big goal was to make the technology "invisible" to users. "I want them to be able to pick up a device and not have it matter if it's a Chromebook, Windows laptop, MacBook or iPad," Perry said. "Their experience is, it just works. They can do what they need to do and they don't have to worry about everything behind the scenes."

Plus, he wanted his techs "to not have to be running around, going, 'Oh, we've just lost the science wing. I don't know if it's a cable or the switch is down.'" His vision: "I want to be able to get a notice: 'Hey, there's a problem brewing at the science wing. Let's schedule somebody to go take a look at that when they have time.'"

Achieving that part of the Project 2020 would require implementing a software-defined network that could do the heavy lifting for IT. For Perry, SDN is all about simplifying and centralizing network management. Sure, there have been a lot of tools and utilities through the years that tackle that, but SDN ties those sources together "in ways that we have never been able to automate before, and allowing the system to identify and react to problems," he explained.

Back in the old days, if a network card went bad and the network was hit by a broadcast storm, IT people had to go and check every single source to figure where it the problem had originated. That could take a lot of time, trying to pin down where it was coming from," Perry said.

The "allure" of SDN is that the "computer can process that information far faster than we can," he noted. "If I have a switch in my high school that suddenly sees a bunch of traffic from a port that it has never seen traffic from before, that switch should be able to say, 'This is not normal; I'm going to cut it off until I can identify what's going on' and do it." That single activity will help protect the rest of the network, while alerting IT through a central console that a problem has been identified, cut off and here's what the system knows about it. "I can follow up and start figuring out what's happening without having my hair on fire."

The SDN Difference

There are three major components to SDN for Carmel's network.

First is that management console, which watches the physical aspects of the network and lets IT know when something has gone down. It also archives and backs up configurations and allows for "zero touch provisioning," in which IT can set up a template on the system, unbox the switch, plug it in, turn it on and have it talk immediately to the console, pull down its own configuration, bring itself up and make itself ready to go. That's a "huge time save," Perry said.

Second is network access control to focus on the user aspects of the network. Right now, that's implemented in a "limited capacity," said Perry. Currently, the NAC can identify devices and device types and some users (those using wireless devices, for the most part). Long-term, however, the NAC "is going to be amped up to identify users wherever they log in, however they log in." At that point the NAC will be able to identify them and pass that data back to the management console to create rules on the fly for that user: "Oh, this is Johnny Bravo in room 28 at the high school. He is in health class, so I need to go tell the web filter to unblock the health-related websites for the next hour, while he's in class." When the student leaves that class, the NAC will identify him or her in the next location, shut down the previous access and return it to the default.

The third piece is the application analytics engine, which is also slated for full deployment during the summer. Currently, what it does is monitor the "health" of the data traveling over the infrastructure. Eventually, when fully up and running, the engine will help with troubleshooting and identification of problems "before our users have a chance to complain."

Recently, that need was brought to light when an on-premise Moodle server started acting up on "old virtual hardware." As Perry explained, "As the students would go to take quizzes, a bunch of them could open it, then there would be five or six in the class that couldn't get in no matter what, and then their session would drop and they would lose the quiz and have to start over. It took us about a week of really watching and testing before we could pin down the problem. It was the server hardware. If we'd had the application analytics at that time, it would have told us within a matter of minutes that the server just couldn't handle the amount of traffic that was coming at it. We moved the server over to new hardware. Problem went away."

The application analytics engine will also be able to handle automated reprioritization of data flow. If there's a sudden spike in Facebook traffic at the stadium or Google traffic from the theater, the engine would recognize that for what it was and reallocate capacity — say, off of the learning management system (because it's after school) to where it was needed.

When that engine and the network access control aspects are finally fully in place, said Perry, the SDN solution will feel far more complete to him. "The network will start becoming self-healing and be able to allocate resources as needed without us having to step in every time and do it manually. That's really important for us."

Choosing a Gear Provider

In the case of the Carmel school system, after doing a search for the optimal gear, Perry recommended Extreme Networks to the district. He chose that company for several reasons. First and foremost, he said, it had the best "cost for performance." Some of the best-known switching vendors came back with "a price tag of anywhere between $1.5 million and $1.75 million to go to an SDN with a single pane of glass management." Extreme's bid was half of that. "I'm a school district. I don’t have buckets of money to throw at well-known network equipment vendors. So, I'm looking for the best deal I can get. Extreme was able to provide that."

Those better-known companies also suggested that if he wanted to save money, he should go with a cloud-based solution. That didn't appeal to him. "We have had issues where our ISP has accidentally disabled our link to the outside world. And being in earthquake country, I have that little bit of paranoia where I want to be able to manage it here, so that if something happens in the broader scheme, I still have access to my equipment. Extreme's management console is something we're able to host on site, and it really has tied everything together."

Extreme's switches gave him some security features that the district only had in a limited fashion before. For example, now IT can set up a profile and have the NAC check against that: Is the anti-virus up to date? Are there weird ports that IT should know about? Is it sending out known malicious traffic or odd pings that suggest the device should be quarantined and examined before allowing full access to the network? Or if somebody opens up malware and the machine starts trying to encrypt network shares, the switch can recognize the abnormality of the traffic and cut it off, isolate it and alert IT for follow-up. "To me, that’s the big difference," said Perry. "We always had to do it manually, by hand, and now we can do it automatically, and much faster."

Finally, the company itself has proven to be a great partner, he said. When the district began putting in new access points, as an example, the IT staff was pushing up against the start of the school year and realized that they needed mounting brackets for particularly unusual ceiling construction, something they hadn't planned for. When Extreme realized there was no way to get the brackets to Carmel through the normal channels in time, a sales engineer headed to the warehouse, loaded them into the back of his car and delivered them personally. "I just don't know any other vendor that would do that," said Perry. "They've been there every step of the way, and we really couldn't do it without them."

The SDN Wow Factor

One aspect of the SDN hasn't played out the way Perry would have predicted. It can't be a mix-and-match set-up and work as seamlessly as network vendor marketing might have you think. While the Extreme management console does provide some oversight of existing Cisco gear in a limited way, he noted, it doesn't have that "deep integration with the switch that would allow the full SDN experience." That was the same with the Cisco and Juniper equipment he was considering. Because a lot of underlying communications really rely on proprietary functionality, to get SDN "to meet its potential, you have to be a monoculture. It has to be all Cisco or all Juniper or all Extreme."

Also, SDN does need to be trained. It's not intelligent out the box. As Perry explained, his team needs to put in rules, to say, "This is what we want. This is what's normal." When the application analytics is fully installed, he's expecting to see more of the "self-learning," where the network itself will figure out what's normal and what's out of the range.

On the other hand, throttling of network intelligence appeals to Perry. "I'm old-school — a bit leery of that." Extreme's full capabilities won't be activated until the district is ready for it. It can be turned on to the point where it will inform IT about what actions it would take, without following through. Then "once we get comfortable with it, then we can set it up in a way that it will go ahead and just do it."

Is SDN living up to its promise? "So far," says Perry. "We've already seen such a huge improvement in our capabilities. Some of it has [come about] so slowly that we didn't notice how much things have improved. But then something will happen, where we'll jump into the tool and it'll tell us exactly what's going on. And then it's that realization of, 'Wow, I couldn't have done this six months ago.'"