Assessment Testing >> In Their Hands
##AUTHORSPLIT##<--->
Handheld devices empower teachers with assessment data they can put to immediate use.
At the Orange County Public School District in Orlando, FL, assessing reading skills among the youngest students used to be quite a process. Relying on rudimentary products such as paper and pencils, the strategy hinged on the bubble sheets teachers administered to students once a year. After teachers scored the exams, they sent them to the district office, where results were scanned, analyzed, and combined to form summary reports. These reports gave teachers information about which students needed extra help, and which subjects were proving to be troublesome. But because the reports took weeks to generate, it was difficult for teachers to use them to better serve the needs of their students.
Everything changed with the implementation of a three-year pilot program that kicked off the 2003-2004 school year. District officials, eager to improve their assessment techniques, turned to Wireless Generation (www.wirelessgeneration.com) to find a way to assess students so that teachers could actually do something with their data. Change came in the form of Palm handheld devices (www.palm.com). Teachers used them to record student performance on a series of questions designed to gauge reading skills. According to G. Lee Baldwin, the district’s senior director of Accountability, Research, and Assessment, improvements were seen virtually overnight. In the first six months, with teachers now able to respond to the dictates of the data, reading scores rose dramatically.
“This technology has eliminated the drudgery of assessment,” Baldwin reports. “We’re assessing our students more accurately, efficiently, and quickly.”
Orange County is not the first district to recognize the benefits of handheld-based formative assessment in grades K-3, when students are too young to take tests, and teachers assess them through observation. Across the country, other trailblazing school districts are getting in on the action too, downsizing assessment efforts into the palms of teachers’ hands. This new trend in assessment mixes software with portable hardware in a way that makes evaluating student skills unobtrusive and easy. Teachers give assessments to students one-on-one, and tap or write on the handhelds to record performance. With up-to-the-minute reporting applications, the technology also enables teachers to tabulate overall performance quickly, providing them with a virtually real-time picture on which students need help, where they need it, and how the teachers can help them best. While dozens of software companies sell tools that they describe as formative assessment (see “Other Options,” below), only a handful of firms sell formative assessment tools specifically for the handheld environment.
Two are New York-based Wireless Generation and Tango Software (www.tangosoftware.com), a division of Liberty Source in Austin, TX. It’s not a fair fight; Wireless Generation plays on the national scale, while Tango controls only a small portion of the Texas market. Still, as both vendors look to grow their market share in the months ahead, the formative assessment industry is emerging as one of the hottest and most exciting areas in educational technology.
Assessing Tech Skills
A new product helps judge the technological proficiency of elementary and middle school students.
Math and literacy aren’t the only skills teachers can evaluate with formative assessment tools; thanks to Portland-based Learning.com (
www.learning.com), educators now can evaluate how deftly students can operate the latest and greatest technologies, too. The vendor recently unveiled TechLiteracy Assessment, an online authentic assessment of the technology aptitude of elementary and middle school students. The product, designed to monitor student progress toward both state and national education technology standards, will be available to all US districts starting in spring 2006. Pricing is $5 per student per year.
Gaston County School Districtin Gastonia, NC, was one of the districts that participated in a recent beta test of the product. Roxie Miller, the district’s assistant director of technology instruction, said recently that it’s a perfect tool for gauging assessment in her district, and should be equally useful for other districts down the road. “TechLiteracy Assessment had just what we were searching for,” she said. “An easy-to-use, online assessment aligned with state standards that assesses student knowledge of computer and technology skills in both multiple-choice and performance formats.”
“Handheld-based formative assessment is important in that you are able to set intermittent measures, or benchmarks, that let you adjust your instruction as you go along,” says Daniel Garcia, assistant superintendent for Curriculum and Instruction at Brownsville Independent School District in Brownsville, TX, which uses the solution from Tango. “The fact that the [technology] allows you to make immediate corrections/adjustments to instruction and add value by re-teaching as needed…is a plus.”
Forming Formative
Assessment in K-12 education is nothing new. Summative assessments—benchmark evaluations conducted at the end of an academic year—have been around for years in the form of big, scary exams. On the other hand, formative assessments, tests that monitor student performance throughout a portion of the curriculum (see “Formative vs. Summative,” p. 34), are relatively new to this decade . In the late 1990s, academic publishers introduced computer-based assessment tests to be administered multiple times during a school year. Since then, interesting trends in elementary assessment have developed: Grades 4-12 have emphasized summative but not so much formative assessments, while grades K-3 have focused more on formative.
Handheld formative assessment technology provides teachers with a virtually real-time picture on which students need help, where they need it, and how the teachers can help best.
Today, the bleeding edge of assessment is written for personal digital assistant (PDA) operating systems and is administered on handheld technology: various devices from Palm (www.palm.com), Hewlett-Packard (www.hp.com), and others. According to Naomi Hupert, senior research associate with the Education Development Center (main.edc.org) in Newton, MA, the approach monitors students on elements they need to master in order to move on to the next level of a particular subject. Hupert says formative assessment is a perfect way for teachers to catch and address gaps in a student’s knowledge or learning, early on. To this end, she adds that the technology provides teachers with the opportunity to teach more effectively.
Formative vs. Summative
The two types of assessment have different aims; one wants to develop student learning, the other wants to measure it. According to Paul Black and Dylan William’s seminal “Inside the Black Box: Raising Standards Through Classroom Assessment” (Phi Delta Kappan, October 1998), formative assessment is a powerful means of improving student learning. It is characterized by the effort of teachers to utilize technology to feed information back to students in ways that enable the students to learn better; it also describes the process of students engaging in a similar, self-reflective effort on their own. Formative assessment is particularly effective for students who have not done well in school, thus narrowing the gap between low and high achievers while raising overall achievement.
In contrast, summative assessment is the attempt to summarize student learning at some point in time; for example, at the end of a course. Most standardized tests are summative. They are not designed to provide the immediate, contextualized feedback useful for helping teacher and student during the learning process. High-quality summative information can, of course, shape how teachers organize their courses or what schools offer their students. The downside, however, is that if a summative assessment is administered too early in the learning process, it discourages teachers and students alike by assuming a knowledge gap.
“If there’s a content area where there is some underlying knowledge, students need to know it and have it available to them,” says Hupert, who also serves as a Reading First evaluator in New Mexico. (Reading First is the US Department of Education’s nationwide effort to enable all students to become successful early readers. See www.ed.gov/programs/readingfirst.) “The process of formatively assessing those students can become a valuable tool for teachers so they can be sure that the baseline of knowledge is being used appropriately by students.”
As Hupert suggests, handheld formative assessment systems don’t only assess student progress; the tests also can provide subtle evaluations of curriculum effectiveness. Here, teachers can use the assessment technology to gauge how well they’re teaching to student needs over time. If, for instance, an October assessment reveals a particular student is at risk of failing to grasp a particular subject, a teacher can reassess that student in December to see if changes in the curriculum have helped. If the student shows progress, the teacher can note what kind of approach to learning works best. If the student demonstrates little to no progress, the teacher knows it’s time to try something new.
Still, handheld-based formative assessment systems are not without a little controversy. For starters, some teachers say that the technology is disruptive, cutting into valuable classroom time that teachers could spend teaching. What’s more, because the latest formative assessments are generating precise and transparent data, many school districts see the tools as panaceas for assessment needs as a whole, and treat the formative tools as if they are more summative than they reallare.
“At the end of the day, the truth is that formative assess-ment is larger than merely assessing more frequently,” says Rick Stiggins, CEO of the Assessment Training Institute (www.assessmentinst.com) in Portland, OR.
The Trailblazer Hands down, Wireless Generation is the market leader in handheld-based formative assessment systems. The company was founded in 2000 with the goal of putting formative assessment into the hands of teachers and getting them away from more cumbersome materials such as desktops and paper. Today, the firm’s mCLASS (mobile classroom assessment) technology is used to assess 10 percent of the K-3 population in 48 states. Half of the Reading First students across the country are being assessed using its software. The company’s products constitute the official Reading First solution in 16 states.
Wireless Generation uses fixed, or preset, assessments to monitor student progress in two main subjects: reading and math. Reading is by far the larger of the two markets, with products specifically designed to assess pre-kindergarten literacy as well as K-3 student performance. This is accomplished via early-reading standards such as Dynamic Indicators of Basic Early Literacy Skills (DIBELS;dibels.uoregon.edu); Phonological Awareness Literacy Screening (PALS; pals.virginia.edu); Texas Primary Reading Inventory (TPRI; www.tpri.org); and El Inventario de Lectura en Espanol de Tejas or Tejas LEE (www.tejaslee.org), which establishes baselines for reading comprehension in Spanish. Math products, set to be released later this year, will cover early numeracy, operations, and all the basics of algebra.
When you get data to teachers in real-time, the entire learning process becomes more responsive in a way that motivates kids to get to the next goal.
Larry Berger, CEO, Wireless Generation
Whatever subject an assessment is evaluating, assessment systems from the predominant vendors work in pretty much the same way. First, a teacher administers a paper-based test and assesses student performance on the test using a handheld computer. When the assessment is complete, the teacher uses the device to look back at student performance as a whole. Next, the teacher puts the handheld device in its cradle and “syncs” all applicable data to the Internet. Finally, the teacher, coaches, and relevant administrators can log in through secure, password-protected access to view a variety of data reports and analyses and use instructional planning tools on a Web site.
“When you get data to teachers in real time, a bunch of things change,” says Larry Berger, CEO of Wireless Generation. “The entire learning process becomes more responsive in a way that motivates kids to get to the next goal.”
This is exactly what has happened at the Orange County PSD, where, at the start of the 2003 school year, district officials signed up for the DIBELS assessment software from Wireless Generation. The product, dubbed mCLASS: DIBELS software, offers teachers a specific K-6 assessment with progress-monitoring questions that meets requirements established by 45 Reading First states. Baldwin, the district’s director of assessment, says teachers in most of Orange County’s Reading First schools are required to utilize the technology four times a year, while those in other schools must use it three times. So far, he adds, reading scores have improved across the board.
OTHER OPTIONS
What else is out there to help your school assess student learning?
Handheld formative assessments aren’t the only types of formative assessments on the market today. Other modern assessment tools come in two additional varieties:
Web-based and software-based.
Web-based tools, known by some as Application Service Providers, are online versions of traditional paper-based tests. Vendors offering this approach include: Tungsten Learning (
www.tungstenlearning.com), SchoolNet (
www.schoolnet.com), Scantron (
www.scantron.com), Renaissance Learning (
www.renlearn.com), Pearson School Systems (
www.pearsonschoolsystems.com), Pearson NCS (
www.pearsonncs.com), Harcourt Assessment (
www.harcourtassessment.com), Princeton Review (
www.review.com), and Edusoft (
www.edusoft.com).
Software-based tools come as programs to be uploaded to a server or to individual computers in a networked classroom. They include products such as Focus on Standards from ETS (
www.ets.org) and state-specific applications from CTB/McGraw-Hill (
www.ctb.com).
With both of these options, teachers set up assessment tests on computers in a lab or at the back of a classroom, and students take the tests at set times, usually hours designated for assessment. In some cases, teachers can tinker with program protocols to establish rules by which different students receive different sets of questions to assess certain skills.
With the right presets, for instance, special needs students in a particular class can receive questions that quiz concepts at a more basic level. Conversely, a system can be programmed to send only the toughest questions to gifted students.
Edmundo Gonzalez, vice president of Sales, Marketing, and Product Management at Software Technology (
www.sti-k12.com) in Mobile, AL, says this is precisely what makes formative assessment so worthwhile. Gonzalez, whose company sells Web-based assessment tools, says that in his business, as is the case in the world of handheld assessments, the ability to make the same assessment different for every child in a class is by far the technology’s biggest benefit.
“The beauty of formative assessments and the technology behind them is that they are entirely customizable depending on a student’s previous performance and standing in the class,” he says. “No matter how you look at it, this is the way educational testing was made to be.”
A Different Approach
Products from Texas-centric Tango Software have yielded similarly encouraging results for the Rio Grande City Certified Independent School District (TX). There, Technology and Instruction Director Vilma Garza says the district has used Tango products since 2002 to facilitate huge jumps in reading ability. During that time, Garza says the district went from having 33 percent of third-graders pass the TPRI reading exam, to a 90 percent pass rate. Rio Grande City CISD teachers have embraced the technology so completely that Garza notes that the tools have changed the fabric of evaluations themselves, making tests something students actually look forward to.
“What once took weeks of manual tabulation [is] now available to us as soon as teachers tap ‘finish’ on the handhelds,” she explains. “Our students love being able to use technology to take their assessment. Can you imagine students being excited about testing?”
Liberty Source, the firm that makes Tango, offers two distinct solutions: Tango RX and Tango Suite. Both of these solutions can be preloaded with benchmarks to assess student compliance with TPRI and Tejas LEE, to name two. The big difference between Tango’s tools and Wireless Generation’s offerings is ap-proach: While Wireless Generation focuses on improving the process of giving fixed, proven assessments and helping teachers and administrators to understand and use the data, Tango emphasizes flexibility by enabling teachers to load any other assessment whatsoever. Edward Barerra, Liberty’s president, says that neither of the Tango solutions differentiates between formative and summative assessment because the company wants to enable teachers to collect any kind of information they want.
This open-ended approach enables teachers to personalize the assessment experience in any way they see fit. Many of them, including Garza at the Rio Grande City CISD, author items for their day-to-day instruction, reword questions, or upload entirely different types of assessments—even summative ones, if they so desire. According to Barrera, the idea behind all of this is to maximize flexibility. Barrera says that by providing educators with these kinds of options within a set of basic parameters, Tango improves upon the likelihood that teachers actually will do something with assessment data once they administer the tests.
“Assessment technology needs to align with a teacher’s personal style of instruction,” he says. “In some instances, that’s not the case, and we need to make sure we help them incorporate our products into the way each and every educator runs [his or her] classroom.”
Down the Road
While both of these formative assessment models work well, experts say that K-12 school districts must overcome some critical challenges surrounding the technology before it enters the mainstream. First, of course, is teaching teachers how to use it. In many districts, teachers are just getting accustomed to having desktop computers in the classrooms, and teaching them how to administer assessments on handheld devices is an entirely different ballgame. In Rio Grande, teachers are required to take workshops on the new technology; at Orange County PSD, the training is ongoing, as 25 teachers run refresher courses in the technology throughout the year.
Rio Grande City teachers have embraced the technology so completely that the tools have changed teh fabric of evaluations themselves, making tests something students actually look forward.
Once teachers learn how to use the handheld technology, perhaps the biggest challenge is getting the educators to actually incorporate it into their curriculum. Stiggins, the CEO at Assessment Training Institute in Portland, says the issue here is contextual communication—finding formative assessment systems that communicate data in context so that teachers know how to use it. Stiggins says the key to ensuring this critical step is a worthwhile reporting system that involves both teachers and students in a continuous process.
“If a formative assessment system generates information about student achievement, it needs to include within the system effective ways to communicate results,” he says. “The best technology and most accurate assessments in the world are wasted if the results aren’t communicated in a way that can be used to improve learning as a whole.”
Wireless Generation’s Berger calls this the “Now What?” syndrome, explaining that once teachers have identified what formative assessment is, and why it’s important, they must learn how to contextualize the data it provides. Berger lauds the use of established and validated assessments that enable educators to make “apples to apples” comparisons of data over time, and says that with these tools teachers can visualize student progress, customize instruction, and help group students according to needs. He adds that his firm currently is working with publishers of instructional programs to integrate its mCLASS assessments products with other curriculum and intervention offerings.
Hupert, the researcher at the Education Development Center, likes this strategy, going so far as to say that no assessment is worthwhile without a valid plan. Educators who prefer a heavy phonics approach to literacy may choose an assessment that emphasizes knowledge of basic phonics in early reading; those educators who prefer to focus on comprehension skills may choose an assessment that includes more emphasis on students reading from texts. Hupert says the bottom line is that a district’s formative assessments shouldn’t be plucked out of a pile and plugged into a program, but rather selected and administered carefully to meet specific achievement goals throughout the year.
“No matter which technology you choose—handhelds, desktops, laptops, whatever—all assessments should reflect student need,” she says. “We need to rethink the way we utilize testing across the educational system, understand that different students require different kinds of teaching, and come to accept that formative assessment is the only way to make the learning process reflect how students really learn.”
Jacob Milner, a regular contributor to this publication, wrote “Warming Up to Wireless” in our November 2005 issue. He is a writer and editor based in northern California.