Assessment | Feature

High-Stakes Online Testing: Coming Soon!

For decades the No. 2 pencil and bubble sheet have ruled the student assessment process. The time has finally come to move those tests online. Here's what you can expect.

• By Dian Schaffhauser
• 06/07/11

High-stakes computer-based testing has been around for more than 10 years, with some states eagerly embracing it and others avoiding it like whooping cough. But the advent of national standards is luring more states into online testing--to the point where districts can no longer assume it won't happen where they are. Almost every (if not every) state in the union has signed on for online high-stakes testing at some point in the future.

Why the sudden rush? According to Linda Rogers, associate secretary for the teaching and learning branch in Delaware's Department of Education, the current political environment values accountability and therefore is willing to support initiatives that place it front and center. Likewise, the existence of the national Common Core initiative is helping drive the work in creating a common assessment.

At the same time, the tools of assessment are becoming more sophisticated. "The stars are all aligning--or maybe the comets are all colliding, depending on your perspective," she says. "[We haven't had] all those things at the same time before, and that's why it hasn't happened before."

The Current State of Testing
Two or three months ago, somewhere in this country--actually in most places--truckloads of pallets stacked with boxes and bundles of pre-labeled test booklets and bubble answer sheets were being delivered from testing companies to school districts all over the country, to be unboxed and unbundled and carefully delivered to select classrooms and particular teachers throughout the schools, where they were counted and accounted for, and issued to students, who spent an hour or two in several sessions stressing over questions and filling in little circles with No. 2 pencils. When the tests were completed, the process was reversed, and so began the waiting for the moment when teachers and principals would finally receive results and know how their kids performed. Some of those people are still waiting for the results, even now, in midsummer, past the point when they can help those same students understand a given topic a little better before they're moved into the next grade level.

With a few exceptions, that's the current state of high-stakes testing in American schools. Those, of course, are the tests that prove to state and federal government agencies--not to mention local communities--how well a given school is doing its job of educating its students. But the testing process doesn't have to be like that. Some states--Virginia and Delaware, to name two veterans--are running those testing operations online. The testing company each state works with delivers tests digitally in a secure and efficient manner. The students sit at computers to answer questions. And the results come back immediately--as soon as the student presses the "submit" button, if the state so chooses. Districts in those states may choose to test students throughout the school year so that teachers can gauge progress and address inadequacies in learning before the last call and students are better prepared for their time in front of the testing screen.

The desire to deliver just-in-time test results was one of the reasons Secretary of Education Arne Duncan took $350 million out of the Race to the Top (RTTT) funding and applied it to the development of what would eventually be known as the Common Core State Standards Initiative for national standards in reading and math.

Duncan's idea was this: Rather than each state individually attacking the problem of raising academic standards and improving the assessment process, states could work together. In 2010, the standards were introduced. Now, with RTTT backing, two groups, the Smarter Balanced Assessment Consortium (SBAC) with 30 states and the Partnership for Assessment of Readiness for College and Careers (PARCC) with 25 states, are developing assessments based on those standards. (Some states are involved in both efforts.) SBAC is promising "state of the art adaptive online exams using open source technology." PARCC is promoting "next-generation" computer-based assessments with "much faster turnaround of results" and "innovations in test items." Both promise to deliver online assessments by the 2014-2015 school year, the same year that the participating states have committed to implementing the new standards.

The Benefits of Online Testing
Almost all advocates will say that the two main benefits of online testing are the two already mentioned: getting rid of all that paper and receiving test scores in a timely manner so as to be instructionally actionable. Yet Virginia's assistant superintendent for technology, career and adult education in the Department of Education, Lan Neugent, can cite numerous additional benefits his state has enjoyed in its move to online testing. As a pioneer in this area, Virginia is the state with all the metaphorical arrows in its back. For the past decade, its districts have been expanding their ability to deliver online standards of learning tests to students in grades 3 through 12. (Having come this far, in fact, Virginia has opted out of the Common Core standards, believing its own to be superior.)

For one, there's the benefit of improved data collection and increased accuracy of student data. "The more people that touch data, the more likelihood there will be mistakes," Neugent notes. "When a student puts their answer on the computer, not as many people touch it. You're getting really good data all the time."

Another plus is that computers can replace the use of a person for certain functions. For example, instead of hiring a person to read to a test-taker with a vision problem, the computer can do the reading or provide a bigger screen or a larger font.

Then there's increased student interest. Delaware's Rogers recently saw some video interviews with students about their eight months of experiences with the online format. "I was mesmerized," she recalls. "These students talked about how much they liked doing it on the computer, how because they test so many times a year, that it doesn't feel like an event anymore--it's an activity. Now they get [test results back] the same day. It means something to them. That was a huge aha for me."

But students aren't the only enthusiasts. District educators who implement online testing become fast fans of the process. In a report on online testing and open source that Educational Testing Service (ETS) issued last year with market research firm Grunwald Associates, a state ed tech director was quoted as saying, "The analogy that we've used is that [moving to online assessment] was like pushing the districts off a cliff--but they found out it was a four-inch drop! I'm aware of no school in our state that has done online testing and wants to go back to paper--once they've tried it, they're sold."

Mastering the Hurdles
Though the benefits are clear, implementing online testing at the district level is not a walk in the park. Between now and the start of the 2014-2015 school year, when most states have committed to adopting the Common Core standards and the consortium-developed assessments, districts and schools have multiple challenges to address in moving to online tests. There are 15,000 districts in the United States, notes Bryan Bleil, director of online and technology implementation for Pearson Education. "And when you think about all the different variables that can come into play, there are almost that many configurations and setups."

In a nutshell, here are challenges districts can expect to face as they transition to online testing.

1. Adequate number of computers. Districts--and states--usually start to make a transition to online assessment by performing a technology survey, basically counting up "how many computers are out there," says Bleil. The count will help the districts figure out how large their testing window needs to be. "If you have a thousand kids and 200 computers, you might need five days to test them all," he explains. Or, you might need to get more machines.
           
2. Adequate bandwidth. To Geoffrey Fletcher, "bandwidth is an even bigger problem than the potential lack of computers." The State Educational Technology Directors Association senior director notes that "if I'm a tech coordinator in a school right now, I'm trying to find every chunk of bandwidth at the cheapest price I can."

While they're out shopping for bandwidth, adds Fred Manno, business technology leader for K-12 testing at ETS, districts need to make sure they're putting in controls on how it's being consumed. "The broadband pipe isn't being regulated at the schools, so kids are doing other things such as social media and video--using capacity that could be leveraged more directly for testing."

Scarcity of bandwidth is one reason that high-stakes online tests are still primarily multiple choice. The more media-rich the test, the more bandwidth each test will need--and the greater the demand on the infrastructure, "especially if you imagine a thousand kids starting at once," notes Bleil

3. Scheduling window. With online testing, schools have a new equation to solve: the number of students testing, over the number of testing days, over the number of school periods, divided by the number of available computers equals the number of potential testing opportunities.

Clearly administrators are up to the math, but the deeper problem is that every computer assigned to test-taking is a computer not being used for instruction. When the test-taking window spans several weeks, observes Bleil, is where the uninformed state could seemingly "punish" its students. "As soon as you talk about a monthlong testing window," he notes, "then you start looking at what can be a significant impact on the instructional use of those same computers. These are challenging kinds of tradeoffs that states make on behalf of their districts."

4. Network security and stability. There are a number of security challenges inherent in online testing: blocking students from outside access; tight controls on what tools (for example, calculators) are available during the test; content filters that might roadblock encrypted test code; and odd, unexplained online glitches that can wreak havoc in a testing atmosphere.

In April 2011, for example, the pilot tests for large groups of Indiana students consistently kicked them offline at least briefly--for minutes at a time--which led some participants to wonder how a full deployment could be possible. Vendor CTB/McGraw-Hill worked "around the clock," according to news reports, to figure out what was causing the interruptions. (At the time of this writing, the cause of the problems hadn't yet been identified.)

Caching can help achieve stability. For tests delivered by ETS, all local PCs connect through a caching server, which has all the tests encrypted. "Nothing is stored on the hard drive," says Manno. "That would be too much of a security risk." When the test is over, school procedures may call for the clearing of the cache, so it disappears totally from the school's systems.

The company experimented with an online-only model of testing a few years ago and gave up on it. Says Manno, "Too many had challenges with it. If a tornado rips through the Midwest and broadband goes down, it can wipe out testing for days. But if we can load the test to the caching server, the students think they're on the internet."

5. Scale. For a high-stakes online test, it's not uncommon to have thousands of students across a district beginning their tests within minutes of each other. "That presents a use case that's almost unique for network administrators," Bleil explains. He compares it to the problem faced by waterworks engineers in dealing with the "flushing surge" that occurs during Super Bowl halftime. "You have to understand your network to understand where and how to put a proxy cache in place, for instance, so you're reducing the amount of data."

6. Staffing. When a testing week rolls around, Marc Baron, chief of performance accountability for the School District of Palm Beach County (FL), makes sure there are technical and test administration staff available. "We try to arm the help desk with as much information as we can, because we don't want to pass schools from one person to another."

During a testing week this past school year, Baron says, an internet service provider's server crashed while students were taking their tests. "We got a large number of calls on that one. A user plugged in something the system wasn't set up to handle, so it broke." As a result, he adds, "Our test administration folks and IT folks had to work closely with the Florida Department of Education and the vendor to work through how best to solve that problem."

7. Budget. Pearson's Bleil will hear IT professionals complain that they're being expected by the states to deliver technology services that they themselves know there's no budget or additional staff time for in their districts. But this, he adds, is a case of IT myopia. "What they typically don't have visibility into is everything already going on inside districts. When you factor that in, this represents a cost savings or a shifting around of cost."

As an example, Bleil cites a large district in Texas with 300 schools where the tests are done on paper. Once those tests are completed by students, the policy is to have somebody in the school box up the papers, load them into a vehicle, and drive them to a central location in the district. "That's a fair amount of work on the school's personnel part," Bleil says. "But it is not work for the tech people to be involved in. They don't know that's happening. They don't see how that goes way when you shift over to technology-based testing."

8. Comparability with paper-based tests.In most states online testing still isn't 100 percent online. "Some students [in a district] are actually taking the test on paper, while others are taking it online," notes Bleil. "That places a pretty significant constraint on the test picture. If you're giving one set of students the richer types of questions while other kids are taking the same test on paper, that does raise issues of comparability and equity." Although he doesn't expect states to wait until every student can take a test online to turn off the paper chase, he does expect that most will wait until the majority of students are testing that way.

9. Overall readiness. Beginning in 2009, pushed along by state legislation, the state of Florida began a transition to more computer-based assessments, continually adding more tests and more grades to the online testing fold. As part of that process, this year Florida had its schools fill out a technical-readiness survey to answer a series of questions about how many students and how many computers they had, and how they could schedule the state-mandated exams.

Here's a common conundrum: Because most schools don't have enough desktop computers to accommodate all testers, they have to rely on laptops. But laptop batteries typically don't last an entire testing day. So they have to be charged during the day--a big challenge, especially in older classrooms whose electrical systems can't handle dozens and dozens of simultaneously charging machines. There are obvious work-arounds (like charging storage stations), but that's the point--there's no clear and easy path to online technical readiness.

So serious is the readiness issue that Virginia has created a "96-hour readiness checklist" for its schools doing online testing. (See "Certifiable in Virginia" sidebar.)

No Time Like the Present
Preparing for online testing isn't like stashing away an emergency kit for a natural disaster, which may or may not hit your area. If you're in a school or district, online testing will eventually seek you out like a cruise missile tracking its target. That means it's time for you to start preparing for the inevitable.

The good news is, the basics of the job are well known. Virginia's Neugent, who has seen the surge of interest in online testing, then a waning, and now the surge again, often says to people, "You really have to start this as a technology initiative, not a testing initiative. What you're doing is building the infrastructure--in a railroad analogy, you're laying the tracks. You can create all the online tests in the world. But if you have no way to move them around or lack the administrative backbone behind it, you're only going to get so far. States that have tried to do this as a testing initiative, they've found out the infrastructure was inadequate."

Which does not mean that IT people are working in a vacuum. Pearson, which recently introduced an online road map for districts implementing online testing, encourages customers to create a close partnership between technology and assessment staff. "These are two groups that have traditionally not had to work very closely together," Bleil says. "The assessment folks know how to manage testing, and the technology folks have the knowledge about the infrastructure and network. But it's when you have both groups working together that you really are able to build the local success you need to solve local issues that might come up."

His advice is to create a readiness team, drawing from both groups so they all understand they're in this together and success is predicated on their collective skill sets. Bleil's colleague, Shilpi Niyogi, executive vice president of strategy and new business development in Pearson's Assessment and Information group, cautions people from thinking that the inevitable is a far ways off. "I think we're going to be surprised by the trajectory we're on now with online testing," she says. "There is a convergence of forces around broadband, around new devices emerging, around policymakers thinking about how we really do need to conquer this problem of the digital divide. It's heating up in a way that's hard to grasp. It might happen faster than we realize."

Certifiable in Virginia For any state or district that still does most of its standardized testing with old-fashioned pencils and bubble sheets, the ultimate goal of online testing of all its students can seem an awfully steep hill to climb. To really understand the multiple hurdles ahead, you only have to look at the challenges Virginia has overcome. Even after 10 years of building up its delivery of those high-stakes exams online--adding up to nearly 2 million--the state still delivers an even larger number on paper. One can't help but ask: What's taking so long? For one thing, high standards. Virginia has found that it's "absolutely essential" for computers to be bulletproof, says Lan Neugent, assistant superintendent for technology, career and adult education in the Virginia Department of Education. "You couldn't have 30 children go into a room to take an online test and have things not working," he points out. To make sure districts met the necessary standards, the state put in place a certification process many years ago. Districts had to certify that their networks could manage the number of tests they were going to deliver. To ensure commitment from the top, the certification had to be signed off by the district superintendent and audited by an outside vendor. Part of the certification process is a "96-hour checklist," which kicks in as test day approaches. "It's like, okay, you did your certification in January. Now it's June. Check it again. Make sure it still works so you don't run into blips and problems," Neugent explains. That list includes sections for both technology people ("Alert your Internet Service Provider to your online…testing window, and also confirm that no scheduled maintenance or outages are planned during that entire window") and assessment people ("Verify that all students testing online have had the opportunity to view the…tutorial."). Neugent estimates that all Virginia high schools are certified, as are 97 percent to 98 percent of middle schools and 70 percent to 80 percent of elementary schools. "It's just a matter of volume," he notes. Until very recently, district participation in online testing in Virginia has been voluntary. Yet each year the number of tests delivered grows by several hundred thousand. "Schools have finally discovered that it's easier and more reliable and efficient to do the online testing," Neugent says. Good thing: The state has now set a goal of having every student in Virginia in every course do online testing by 2014--the same deadline as set by the Common Core consortia.