April 2007 — News
Print this article | Email this articleClick here to receive your FREE subscription to T.H.E. Journal
DOE Software Study: Are the Numbers Flawed?
4/6/2007—The United States Department of Education (ED) Thursday released the results of a study measuring the impact of math and reading software on student achievement. The results? "Test scores were not significantly higher in classrooms using selected reading and mathematics software products." This didn't sit well with organizations (or individuals) in education technology. Several responded to the study, denouncing the research as flawed or cautioning careful scrutiny of the data.
The report for Congress released by ED, titled Effectiveness of Reading and Mathematics Software Products, was based on research that commenced in 2003 in conjunction with Mathematica Policy Research Inc. (MPR) and SRI International. Participants included 33 districts, 132 schools, and 439 teachers using 16 different software products, which were selected based on recommendations by "expert review panels" and the study team. Products covered first- and fourth-grade reading, sixth-grade math, and algebra. There were 9,424 students involved in the sample who were tested in 2004 and 2005.
In a joint statement issued Thursday, the Consortium for School Networking (CoSN), the International Society for Technology in Education (ISTE), and the State Educational Technology Directors Association (SETDA) cautioned readers of the report "to scrutinize the findings carefully, as even [ED] states that the study 'was not designed to assess the effectiveness of educational technology across its entire spectrum of uses.'" These organizations also noted that there are several "scientifically-based studies" from independent research organizations funded by ED that show technology's positive impact on learning and instruction.
But essentially, said critics of the study, ED didn't take into account the critical factors of proper implementation and curriculum integration, professional development for teachers, planning, or infrastructure issues, among others.
The Software & Information Industry Association (SIIA) also released a statement on the research. "As this study recognizes, proper implementation of education software is essential for success. Unfortunately, it appears the study itself may not have adequately accounted for this key factor, leading to results that do not accurately represent the role and impact of technology in education."
The results of the study were not all net-zero. But, on the whole, each of the four categories in the study showed no statistically significant differences between the test groups using software and the control groups. This runs counter to other studies, said critics, that have shown the positive impact technology has had on education.
"Across the nation, educators are using a range of learning technologies to transform teaching and learning and improve student achievement. We have solid evidence that these efforts are working," said Mary Ann Wolf, Executive Director of SETDA, in the joint statement.
"It is important to remember that educational software, like textbooks, is only one tool in the learning process. Neither can be a substitute for well- trained teachers, leadership, and parental involvement," said Keith Krueger, CEO of CoSN, in the same joint statement release.
