Extreme Makeover: Achieving a Solid Core Infrastructure
If your IT staff expends most of its energy running in place just to keep up with support demands, it may be time for a data center makeover. Upgrading to new components such as server virtualization and power and cooling management allows you to work more efficiently and strengthens your ability to support your district’s educational mission.
The day was July 15, 2008. The clock had just begun ticking for Eileen Hershman, educational research and technology coordinator for Upper Darby School District in Drexel Hill, PA. Hershman’s name was attached to 14 of the 18 major goals set out in the district’s newly adopted three-year plan for educational technology. But Upper Darby had a disconnected set of systems on its hands. As they tend to do, users kept demanding new technologies at the same time that the technical services organization was overwhelmed. A primary source of the trouble was a networking infrastructure and outdated data center that hindered IT’s efforts to keep up.
The data center for the district consisted of a collection of servers——each running its own application——with no central means of management. Without a directory such as Active Directory in place, there was no way to manage users or their security roles. Storage and backup was haphazard. Climate control in the data center worked erratically. There was no way to manage switches or perform server or workstation updates or repairs remotely. When somebody wanted a new application, IT would have to order a new server and get it up and running——a two-month process——before the software could be deployed. “We needed to work smarter,” Hershman concludes. “We needed ways to streamline our operations.”
Strengthening those aspects of the data center that support the efficient running of most other parts of your organization to improve performance requires a four-step process: planning, buying, rebuilding, and settling in for the long haul. If you feel like your IT team is struggling to keep up in supporting your schools’ tech initiatives, perhaps it’s time to take lessons from others that have been through the exercises already——including both Upper Darby, which refurbished its existing operation, and Lee County Public Schools in Florida, which moved into a new data center.
Step 1: Planning
The work for Upper Darby really began when the district brought together a team of people to help develop the three-year technology plan for the district, which has about 11,700 students. The district has money allocated for technology every year. In spring 2009 the team agreed that the money for the current school year would go to the networking and data center project “instead of buying more devices,” Hershman recalls. Those initial plans included hiring a network expert to assess the entire network and advise the district on what forms the network infrastructure improvements should take. The district hired Terrence Norko, now supervisor of network operations.
Norko’s s recommendations focused primarily on
increasing reliability, including addressing environmental and security control issues in the data center and applying standards to networking equipment and operations. Another strategic goal was to achieve single user sign on. “We needed Active Directory or something similar to handle the credentials and tie everything together,” he explains.
A plan was also at the heart of the data center makeover for Lee County Public Schools in Fort Myers, FL. But this district, which has about 80,000 students and 10,000 employees at its 100 campuses, took advantage of a move into a new central office to map out its network and data center infrastructure upgrade. Up to that point, the district’s administration and other operations were spread around the county. “I had inherited a pretty poor central infrastructure,” explains Dwayne Alton, director of IT support. “The data center didn’t have sufficient electrical protection. We had no redundant networking in that facility at all. Our cooling systems were pretty much maxed out.”
A major part of Alton’s plan was to justify expenditures related to rebuilding the data center. “We couldn’t use the old facility as a model,” Alton says. “We needed to eliminate that as the frame of reference for what we should be doing in the new facility.”
Major priorities for Lee County were scalability, flexibility, and redundancy. “We wanted flexibility in design,” Alton says. “So if we had to scale rapidly or migrate from different types of equipment, we’d have the capability in this room. We were looking for a fault tolerance that we didn’t have before.” That included network, power, and cooling redundancy, as well as survivability of the physical building, which is located in an infamous weather corridor known as “Hurricane Alley.”
Step 2: Bidding
Lee County’s Alton was confident in his team’s ability and skill levels to handle the network side of the infrastructure makeover. Because he was less confident about the power and cooling aspects of the data center project, he brought in a private consultant to help put together a request for proposal to lay out power distribution needs as well as the thermal design for achieving optimal cooling and airflow in the facility.
The district’s RFP, with a budget of $500,000, generated proposals from several companies, and a short list of three companies was brought in to make presentations. The winner——CDW-G——stood out, says Alton. “Not only did everybody that came down on the team know the proposal, but we had the president of each of the sub-contractors there. Having the people there directly responsible for the various parts of the project helped a lot.”
At Upper Darby, network supervisor Norko put together an RFP that was specific down to part numbers in some cases. The proposal included replacing all of the server, switch, and storage hardware on the network and introducing server virtualization. (See “The Virtues of Virtualization.”) “We were able to compare very similar devices to see who would give us the best price and also the best value.”
With a couple of exceptions, the companies that responded to the RFP tended either to ignore the specific details of the proposal and recommend what they had in their line-up or offer technology that wasn’t the latest. As stand-outs only CDW-G could deliver VMware 4, which had just been released, and only HP could supply the latest storage area network technology sought by the district.
Step 3: Rebuilding
During the configuration and deployment of its new data center, the Lee County IT crew had to work around the general construction to convert the district’s newly acquired shopping mall into an office building. That required careful planning and weekly phone meetings among the IT project manager, the project manager from CDW-G, and a project manager from the construction department.
The actual move into the new facility reads like a clipboard exercise. While construction was still underway, Alton’s team placed a core switch into the new facility then bridged it with the existing network to keep services up and running in both old and new sites. Then, the weekend of the move, operations shut down at 5 p.m.
A professional moving company hauled the data center equipment to the new location——switches, servers, telephone hardware, the mainframe, and storage systems. On the other end, internal IT and CDW-G staff stood by, waiting for the gear to arrive. Working with the wiring contractor, the hardware was racked and connected. Over subsequent weeks the remaining 40 administrative departments——payroll, personnel, student assignment, and so on——were moved, two per weekend.
At Upper Darby, the remodeled data center would be in the same physical space as the old, which wasn’t very large——about 140 square feet. Since the district had a summer school running and other work being done elsewhere in the building, shutting the network down to do the upgrade wasn’t an option.
After hours, the IT group came in to relocate the servers that had to stay on and put in temporary cabling to keep those machines connected to the fiber data communications backbone. The servers that weren’t essential were shut down and removed. Other equipment was also moved out to make room to fit new racks. The room required painting, new ceiling tiles, lighting, fans, an air conditioning system, and cat5 and cat6 wiring. “We had to cover the equipment as best we could, but still allow it to breathe, because we had to keep it cool,” Norko says.
To expedite installation of the new equipment, IT pre-assembled racks and pre-terminated cables. Once the demolition work was done and the room was classified as “clean,” the team installed its new VMware equipment. Then, Norko explains, “We were able to take the old equipment and transfer the contents of hard drives from the old physical servers into the new virtual environment and then decommission the old hardware.”
Step 4: Settling in for the Long Haul
Both districts are rightfully proud of the outcomes of their core strengthening efforts.
“It’s like night and day,” says Upper Darby’s Hershman. “We changed from an old, dusty, hot wiring closet to a state-of-the-art air conditioned, security monitored, environmentally controlled and managed virtual environment.” No more waiting two months to deploy a new application. When computers need to be updated, most of the time it can be done in minutes remotely. “Our repairs are down. Our technicians are working with users instead of fixing computers. We can manage more equipment with fewer people.”
“Everything was as overly designed as we could make it,” Upper Darby’s network manager admits. “Every piece of equipment is now completely logged automatically,” Norko says. “Security is enhanced and data center access is controlled and monitored. The racks are physically locked. Even if you gain access to the room without an access card, you can’t get into the racks, which are audit-logged. Environmental controls are in place for remote monitoring for smoke, fire, leaks, moisture, water, security. We have security cameras in there that snap a picture and record whenever there’s motion in the room.”
All power is monitored and managed. If the ambient room temperature were to fail, IT would get instant notification. If a device stops drawing power when it shouldn’t, IT will be alerted. If a device hangs after hours, a member of IT can restart the device through a secure interface.
“Overall reliability has gone way up,” Norko says. “And user confidence in using our technology has gone up too.”
Alton cites multiple benefits from the data center makeover at Lee County as well. “We have much higher reliability,” he says. “The network, power and air conditioning are now redundant. The reliability and scalability are much better. We don’t have the issues we used to have because we’re monitoring everything inside and outside the box. We monitor power consumption in real time. That’s incredibly important with any infrastructure of this size.”
“The ability to take advantage of economies of scale by centralizing your applications is huge,” Alton proclaims. “We’ve literally offset all the cost of new servers in what we’ve saved by not replacing servers at the schools. All instructional software and assessment systems have been centralized to run out the data center. An application like Renaissance is now on two servers in the data center rather than 80 servers in 80 schools.”
Finally, the data center itself is hardened——as a building within a building, including concrete walls and ceiling. “It’s designed to withstand a Category 5 hurricane even if the rest of the building is damaged,” Alton says.
The Path to a Hardened Core
Lee County’s Alton points to planning as the key to
his district’s success with its data center overhaul.
“We made adjustments where there were issues, but we didn’t vary from our plan,” he says. “We did so much prep work and testing ahead of time. And that’s really important. As much as people hate to admit it, the IT department tends to be the heart of their operations. So when you’re moving the heart of your operations, you don’t have a lot of wiggle room. Planning and preparation are really important.”
Upper Darby’s Hershman also sees the value of planning, particularly when it results in a formal document. “Having this strategic plan has really helped move things along. That includes getting the right people——the school board, the superintendent, the business manager, the facilities department——to buy into it and say, ‘This is what we’re doing. This is why we’re doing it. It makes good business sense.’”
Now, a year and a half before the current plan is set to expire, the Pennsylvania district has already begun work on its next three-year strategic plan. “We’re big on planning and thinking and doing,” Hershman says. “But really, we’re just meeting the user needs.”
THE VIRTUES OF VIRTUALIZATION
Virtualization really can help a school district do more with less.
At Lee County Public Schools in Fort Myers, FL, Dwayne Alton, director of IT support, oversaw relocation of network operations, including 70 servers, into a new data center. Implementing VMware on that equipment as part of the move enabled his IT team to reduce the count to 60 physical servers that now run about 170 logical servers. But that’s only part of the story. The district had traditionally counted on schools to run their own servers too. By going virtual, Lee County has also been able to centralize IT operations, pulling applications, storage services, and more out of individual schools. As a result, the district has been able to reduce its total server footprint across the district by 60 percent. “We’ve literally dropped 300 servers out of the county since we’ve moved here,” Alton says. Because most schools no longer have their own servers, the district has also reduced its fi eld support calls by 80 percent and upgraded the caliber of equipment that remains in the schools.
Server virtualization has become an essential ingredient in a data center overhaul for school districts. After all, the use of products from VMware, Citrix, Microsoft and other companies enables an organization to reduce the number of servers that need to be operated and supported and allows the staff to shift technology resources on the fl y. Multiple operating systems can run on the same physical box. Enterprise applications no longer require a dedicated server. But the advantages go beyond those obvious ones, as two school districts—Lee County and Upper Darby School District in Drexel Hill, PA—illustrate.
POWER MANAGEMENT
When Lee County implemented its new data center (see “Extreme Makeover: Achieving a Solid Core Infrastructure”), the district was able to take on the responsibility for data backup for schools, a job that had previously been handled by the schools themselves. “We provided them servers with redundant storage to do backup on site,” Alton says. After the move, Alton’steam was able to structure the virtual network to assume real-time replication of all critical data from the schools to the district’s storage area network. Another aspect of virtualization that Alton appreciates is the ability to scale power consumption based on system usage. “A lot of people understand the value of virtualization when it comes to stability and scalability,” he points out. “But another thing it gives us is the ability to scale our power consumption based on our system usage.” During off -peak hours the VMware software monitors performance patterns and knows to shut down some servers to reduce power consumption.
DISAPPEARING REPAIRS
At Upper Darby virtualization has provided a less expensive means to handle disaster recovery, which existed on separate systems before the district upgraded its network infrastructure and refurbished its data center.
Originally Eileen Hershman, educational research and technology coordinator, and Terrence Norko, supervisor of network operations, expected to create a secondary disaster recovery site to mirror operations in the data center as a means to back up critical data. The data center, which underwent renovation in the summer of 2009, includes new servers running VMware 4.0. “We found that we don’t need to have as large a [plan] for disaster recovery due to the nature of virtualization software,” Hershman says. Whereas previously storage was “all over the place,” that’s been redesigned to fi t into two separate storage clusters, one for server data storage and the other for user data storage. That wouldn’t have been possible without the server consolidation that could take place through virtualization.
So, in place of a full-blown secondary site to act as a replacement data center in the event of a disaster, the district has contracted with its internet service provider to house some new servers. Once that’s in place, IT simply needs to replicate the critical data to a few computers.
Also, server virtualization allows for virtual machines to shift from one logical server in a cluster to another. “If one of those physical hosts fails, or we need to do maintenance, the users don’t even notice. It’s completely managed and automated,” Norko says. “VMware will take the virtual servers running on that physical piece of hardware and push them onto the ones that are still running.”
With the new IT infrastructure in place, the district rolled out 200 virtualized workstations for users using thin clients from HP. The result has been a dramatic drop in repairs.
“We took all the iMacs out of our alternative education program and put in thin clients,” Hershman says. “Our repair budget went to $1,500 budgeted for last year to zero this year. We haven’t done one repair in that room.” As she explains, the users were working with web-based curriculum, which only required the use of a browser. Since the Macs were “old and slow,” they were constantly breaking and needing parts. “This consumed a lot of time for students, teachers, and for the technicians, as they had to diagnose the problem, order the parts and wait for the parts in order to get the machines up and running. [By putting in] new thin clients, students can access the curriculum without interruption, and technicians’ time is not consumed with getting the machines fi xed.”
When somebody wants to make a change to the software running on those virtual machines, Norko can make it on the terminal servers in the data center and have it applied to all 85 workstations. “We don’t have to spend days building a new image and deploying it. We can do it in a manner of minutes in most cases.”
The thin clients deployed as a result of the new infrastructure are much cheaper than a desktop computer --- less than half the price, the district says. “We also can deploy new applications without having to purchase new servers and incur the expense of power, cooling, and space,” Norko adds. “Furthermore, we’re able to use technicians’ time more strategically by deploying new applications, working with end users, etc. rather than having technicians doing repairs, upgrades, or reimaging.”
The changes made at Upper Darby were “drastic,” Norko observes. “We went from a very physical environment— one server running one application—to a completely virtual environment, where there’s not a physical box for the art department or a physical box for the English department. Now it’s inside a box somewhere and, to be honest, I don’t know what box it’s in, because it’s managing itself; it’s moving around; it’s automated.”
The move to virtualization came with a “scary price tag,” Norko points out. “But ROI is very quick in this type of environment.” He proves his point by tallying the advantages: “Using staff more strategically to manage a greater number of software applications, quicker end user turnaround time for support, with fewer requests and money spent on repairs.” As he concludes, “We have a small IT staff for the size of the district, yet, we can handle more applications, with better management of resources.”