Security | Feature
How to Survive a Cyberattack
Here's how a North Carolina district responded to a denial-of-service attack that came from one of its own schools.
| You can see more great feature articles in the latest issue of our monthly digital edition. |
At 7:45 a.m. on Monday, April 8, 2013, 23,000 network users in the Rowan-Salisbury School System’s 35 schools were accessing their Web-based curriculum resources and administrative applications when suddenly all Internet connectivity stopped. The outage lasted for about an hour. Teachers had to quickly switch their lessons to a Plan B, since most had components that required Internet access.
Internet connectivity returned briefly, but suddenly went down again for another hour. The Internet would go down for a third time before school ended.
The Internet outages weren’t due to hardware failures, Internet provider outages or network maintenance. The school district was under attack — specifically, we were experiencing a distributed denial-of-service attack, or DDoS. DDoS attacks, one of the most common forms of cyberattack, are designed to overwhelm a targeted IP address with numerous network requests in order to interrupt or suspend network service.
Each morning for the next four days, the district lost Internet connectivity around 7:45 a.m. The Internet would remain down for about an hour and would go down two more times each day. During the next three weeks, the school system was attacked multiple times a day on eight different days. Each attack caused a loss of Internet connectivity. A great frustration among students and staff ensued. Teaching and learning was being significantly impacted.
Identifying the Problem
The school system’s Internet connection was a 1 Gb connection. It was provided by the state of North Carolina as part of the North Carolina Research and Education Network (NCREN). This network serves all the state’s public education facilities, as well as many private colleges and universities and the NC Office of Information Technology. The NCREN system is managed by MCNC, a nonprofit organization. During the three weeks of Internet outages, numerous conversations took place between the MCNC staff and the district’s technology department. It wasn’t until the third day of Internet outages that MCNC identified DDoS attacks as the cause of the outages. Data analysis showed that the district’s network was being attacked with as much as 6.4 Gb per second. The attack was focused on the public IP of the district’s firewall. The purpose of the attack was to disrupt Internet connectivity to and from the school district.
Unfortunately, MCNC wasn’t able to stop the DDoS attack at the perimeter of the NCREN Internet connection. Stopping the DDoS attack at the perimeter is essential to mitigate its effect on a network. Instead, MCNC implemented several measures to try to mitigate the attacks. These helped preserve connectivity for the other organizations sharing the network connection line with Rowan-Salisbury, but Internet connectivity for the district continued to be disrupted.
The district got some help from a retired 3Com/HP network engineer, Larry Tolbert. He had over 30 years of network engineering experience and was very familiar with Rowan Salisbury’s network. After the district leveraged several of its infrastructure components to obtain additional information about the DDoS attack and to block the attack at the perimeter of the district’s network, Tolbert configured the district’s TippingPoint Intrusion Protection System (IPS) to capture packet data of future attacks for analysis. He also configured the IPS to prevent future attacks from reaching the district’s firewall and to send alerts to the technology department at the start of any future attack.
As we discovered, the DDoS attacks were coming from hundreds of malware-infected, remotely controlled computers located all over the world. This made it difficult to determine the responsible person, but the district needed to try.
Calling the Cops
The next step was to get help from local law enforcement. Sgt. Detective Roger Hosey, who worked with the FBI’s Cyber Taskforce and had previously worked as a school district technology technician, knew the system and its network.
The attacks continued for a fourth week, and on that Wednesday, Sgt. Detective Hosey and another officer visited each of the district’s six high schools to interview staff members. They suspected a student might be responsible for the DDoS attacks. The staff interviews didn’t reveal any specific leads, but they did raise awareness within the schools that law enforcement was investigating the Internet issues.
Sgt. Detective Hosey suggested assigning each high school a specific public IP and changing the firewall’s public IP again. The changes were designed to limit the impact on the system’s Internet connectivity if an attack was focused on an individual high school’s public IP. Also, giving each school its own IP might help make a connection between the person responsible for the attack and the high school being attacked.
After law enforcement visited the high schools, DDoS attacks stopped for the next 15 days. On May 16, though, an attack was made on one of the high schools. Sgt. Detective Hosey believed the person responsible for the attack was using an online resource to determine the new public IP assigned to the high school.
An Attack From Within
The district’s Palo Alto Networks firewall contained extensive reporting data, which showed that two computers in a Career & Technical Education (CTE) computer lab at a high school had accessed the website, IP Chicken to find the public IP of the attacked high school. A four-hour review of the firewall’s logs also showed that the same two computers accessed a website, xboot.net, that allows individuals to buy, schedule and launch DDoS attacks on any public IP address that the user designates. Sgt. Detective Hosey, the assistant superintendent of administration, and I met with the CTE computer lab teacher to try to determine which students had been using the two identified computers.
Unfortunately, the teacher had not used good classroom management and technology supervision practices on the day when the attack had been launched, and was only able to suggest a couple of students she believed “may” have been using the two computers. We asked the teacher to make sure she employed good classroom management and technology supervision practices in the future, and we asked her to keep the meeting confidential. Somehow, though, that meeting information was leaked at the school, and students in the class were made aware of the situation.
We configured the district’s IPS system to send alerts if anyone accessed IP Chicken or xboot.net, and waited for the next attack. None came until November 2013, when an alert indicated that someone had visited xboot.net. Firewall and IPS logs showed that a personal iOS device had been used to access xboot.net via the wireless network at the same high school that was linked to the previous school year’s DDoS attack.
Aerohive access points provided wireless connectivity in the schools. These devices provide detailed reporting, and the district was able to determine that the user accessing the DDoS attack website was in the high school’s gym. Calls to the high school’s administration and school resource officer resulted in the user being located in the gym. The browser history on the device, an iPod touch, confirmed that it had accessed the DDoS website — but no attack had been initiated. The student was one of the two that the CTE computer lab teacher had suspected. The school’s administration and local law enforcement took charge of the situation.
Since the initial DDoS attack, MCNC has installed a chief security officer and is looking at solutions that will safeguard the network against events like this at the edge in order to keep such attacks outside the main network. The school district’s IPS alerts remain in place, waiting for the next attack to occur.