Making the Case: Research Efforts on Educational Technology
##AUTHORSPLIT##<--->
Historically, very little, if any, research that meets the scientifically based standards as defined by the No Child Left Behind Act has been conducted on the effectiveness of educational technology. Clearly, the educational technology community must invest in research and evaluation studies to better guide the effective use of the investment, as well as to demonstrate to policy-makers the impact on teaching and learning.
In an effort to address the need, the U.S. Department of Education is investing more than $56 million to study the conditions and practices under which technology is used to document its impact on student performance. The results of these efforts should enable the educational technology community to be in the forefront of evidence-based research on educational practices involving technology.
Some of the federal funds are supporting studies at the national and state levels. In addition, the Education Department's Institute for Education Sciences (IES) is funding technology research at independent research and development organizations and at institutions of higher learning nationwide. Below are brief descriptions of the various federally funded research efforts that are examining the impact of technology on student achievement, professional development and other educational outcomes.
National-Level Studies
National Study Regarding the Effectiveness of Educational Technology. The NCLB Act of 2001 calls for a five-year, $15 million study of the effects of educational technology, using rigorous scientifically based methodologies. In October 2002, the Education Department began working with Mathematica Policy Research Inc. and its partners, the American Institutes for Research and the Education Development Center Inc., to identify issues confronting a national study of technology effectiveness and to develop designs for the study.
A key part of the design effort engaged a panel of outside experts on educational technology, educational policy and research methodology to help identify important questions to be addressed in the study and to suggest possible approaches for answering them. Designing such a study is a significant undertaking. Since no study of educational technology has used experimental methods on such a large scale, important considerations must be addressed, including how the study's questions should be focused, how to structure the design for measurement and resource efficiency, and how to collaborate with schools and districts. Other considerations include the rapid innovations in computer technology, the changes in the educational policy context created by NCLB, and the goal of ensuring that knowledge from the study is immediately useful for contributing schools and teachers.
In May 2002, the design team provided the following recommendations:
- Question: What is educational technology?
Recommendation 1: Examine technology applications designed to support teaching and learning.
Recommendation 2: Use a public submission process to identify technology applications to study.
- Question: What is "effective"?
Recommendation 3: Use experimental designs to measure effects.
Recommendation 4: Study the effects of technology applications for schools or teachers who don't currently use the applications but are interested in using them.
Recommendation 5: Design the study to detect "moderate" to "large" effects of technology applications.
- Question: What kinds of students?
Recommendation 6: Study the effects of technology applications for K-12 students.
Recommendation 7: Study the effects of technology applications for schools that are eligible to receive Title I funds.
- Question: What is academic achievement?
Recommendation 8: Study the effects of technology applications on student academic achievement as measured by commonly used standardized tests, and collect data on other academic indicators to provide a fuller picture.
Recommendation 9: Study the effects of technology applications that support instruction in reading and mathematics.
In September 2003, Mathematica Policy and their subcontractor, SRI International, were awarded a three-year, $10 million contract to conduct the actual study. This team brings together expertise with randomized assignment research and the topic of educational technology. The team will immediately begin the work of identifying the technology interventions and schools that will participate in the study.
National Educational Technology Trends Study (NETTS). NETTS will examine program implementation in schools receiving federal Enhancing Education Through Technology (EETT) grants authorized under NCLB, with a particular emphasis on understanding how and to what extent the EETT program helps further the goals of NCLB. In addition, the study will collect data relevant to EETT program performance as detailed in the Education Department's Strategic Plan.
Evaluation questions:
1. How do states differ in their plans and strategies for using EETT funds?
2. What types of entities are receiving EETT funds (a) under the competitive program and (b) under the formula grant program?
3. How are subgrantees using EETT funds?
4. Are school uses of EETT funds supporting program goals?
a. Is the EETT program helping to close the gap between high- and low-poverty schools regarding students' and teachers' actual access to technology?
b. Is the EETT program supporting teachers, principals and school administrators in effective integration of technology into curricula and instruction?
NETTS will also investigate how access, support for, and use of educational technology vary by school poverty rates and other key factors, as well as whether EETT funds have improved integration of technology in districts of high poverty. The study is sponsored by the Education Department's Policy and Program Studies Service, and is being conducted by SRI's Center for Technology in Learning, American Institutes for Research and the Urban Institute.
State-Level Studies
Evaluating State Educational Technology Program (ESETP). ESETP is a federal grant program designed to increase the capacity of states to design, conduct and procure high-quality evaluations of educational technology. The grant competition provides more than $15 million over the next three years for states to:
1. Plan and conduct a scientifically based evaluation of an educational intervention that uses technology applications to increase student achievement in one or more core academic subjects.
2. Test and document the methods, practices and instruments used to assess the impact of the intervention on student achievement.
3. Make documented information about the evaluation plan and its implementation available to other states.
The Education Department, through its School Support and Technology Programs Office and its Office of Educational Technology, made grant awards to 10 State Education Agencies. The focus of the studies is described in brief below.
State Educational Technology Evaluation Grants
The U.S. Education Department, through the School Support and Technology Programs Office and the Office of Educational Technology, made grant awards to 10 State Education Agencies. Descriptions of the studies, including the scientifically based methodologies on which the studies are based, are provided below.
Arkansas Department of Education
Evaluation of the EAST (Environmental and Spatial Technology) Initiative
$1.8 million over three years
Online: www.eastproject.org
This project will assess the nature, quality and intensity of EAST program implementation strategies and processes. It will also evaluate their relative outcomes on teachers' attitudes, classroom practices and content knowledge, as well as on students' attitudes, skills and achievements. EAST involves the creation of interdisciplinary school-based technology labs that promote student intellectual growth and technology skills acquisition, as well as teacher training on facilitating student learning through service projects and teamwork. The study will involve 120 projects serving 9,000 students (55% rural, 25% suburban and 20% urban setting). Study results will yield deeper insights into specific participant, environmental and program characteristics that appear to influence student outcomes. In addition, an evaluation sustainability study will assess the extent to which the project's dissemination and aptitude-building activities are serving the capacity of Arkansas and other states to plan, conduct and procure high-quality evaluations.
Iowa Department of Education
Using Technology to Support the Scaling-Up of the Iowa Professional Development Model
$1.9 million over three years
Online: www.state.ia.us/educate
This project will use the Iowa-adopted professional development model, which is based on best practices, as the basis for scaling up an educational intervention system using experimental, quasi-experimental and randomized classroom trials. It will seek to demonstrate that scientifically based teacher training on best practices using technology must be causally linked to the implementation of those practices in the classroom in order for the impact to be observed in student achievement in math and reading. This multiagency, statewide field-research effort will focus on reading and mathematics instructional practice in the sixth, seventh and eighth grades in 150 public school districts, including 43 high-need districts as defined by the NCLB Act. The project will yield a research model for identifying and scaling up teacher training on best practices.
Maine Department of Education
The Impact of Teachers' Professional Development on the Mathematics Achievement of Low-Performing Rural Students in Technology-Rich Classrooms
$1.9 million over three years
Online: www.state.me.us/education
The Maine evaluation uses an experimental design with randomized assignment of schools to investigate the impact of intensive, multifaceted professional development on teacher classroom practices, as well as the impact of student and teacher use of technology to enhance mathematics learning and student mathematics achievement. The study will focus on seventh- and eighth-grade students in schools that serve low-income rural communities and that have shown low performance in eighth-grade mathematics. The study promises to contribute to the research-based knowledge of effective practices in mathematics education and technology integration, ubiquitous computing, professional development, and education in low-income rural schools.
North Carolina Department of Public Instruction
LANCET: Looking at North Carolina Educational Technology
$1.5 million over three years
Online: www.ncpublicschools.org/news/03-04/103103.1.html
The LANCET project will use experimental, quasi-experimental and case study designs to study the implementation of the state's IMPACT professional development model, as well as its effects on schools, teaching practices and student achievement. The project will develop and assess strategies for building the capacity for educators statewide to collect, analyze and use evaluation data for making decisions about technology programs, projects and practices. It will also disseminate the strategies, methods, instruments and protocols used in and resulting from the project.
Pennsylvania Department of Education
Evaluation of Student and Parent Access Through Recycled Computers (eSPARC)
$1.8 million over three years
Online: www.pde.state.pa.us/ed_tech/cwp/view.asp?Q=100397&A=169
The eSPARC study project seeks to develop and test an evaluation model that can be used by local and state education agencies to measure the impact of educational technology initiatives. The project will randomly assign recycled computers to a sample of 400 fifth-grade students and their families. The study will then assess whether and how in-home computer and Internet access impact students and patents. The study will also produce and disseminate research methods and tools that can be used to measure the impact of technology initiatives across program areas.
Tennessee Department of Education
The Tennessee EdTech Accountability Model (TEAM)
$1.7 million over three years
Online: www.state.tn.us/education/acctteam.htm
This project will measure the effectiveness of an intervention to prepare school-based technology coaches to work with teachers on methods of aligning technology use to the delivery of the curriculum by using instructional materials that foster increased student achievement. The project will measure effectiveness of the intervention in 37 schools; develop a replicable, validated evaluation protocol for use in all schools, and disseminate the results and instruments nationally.
Texas Education Agency
Evaluation of the Texas Technology Immersion Pilot (eTxTIP)
$1.9 million over three years
Online: www.tea.state.tx.us/
TXTIP is a state-mandated technology immersion pilot that seeks to increase student achievement by providing each student with a wireless mobile computing device, software, as well as online and other learning resources. The TxTIP evaluation will test the effectiveness of technology immersion in increasing middle school students' achievement in core academic subjects, technology proficiency, attitudes and attendance. The evaluation will also test its effect on the school environment, personnel, as well as on parent and community partnerships. Approximately 38,000 students and 2,700 teachers in about 60 randomly assigned middle schools will participate in the evaluation.
West Virginia Department of Education
ED PACE: Educational Development for Planning and Conducting Evaluations
$1.4 million over three years
Online: http://wvde.state.wv.us
ED PACE will employ a quasi-experimental design with experimental elements to assess student achievement in virtual foreign language courses as compared to the achievement of students in classroom-based foreign language courses. Over the course of three years, the project will generate three scientifically based research models (a summative research model, a formative research model and an action research model) that can be replicated in other settings used at local, state and national levels to measure the impact of other technology-enhanced interventions on student achievement and to validate their effectiveness.
West Virginia Department of Education
The Evaluation of West Virginia's Enhancing Education Through Technology Model School Project
$1.3 million over three years
Online: http://access.k12.wv.us
The project will assess the outcomes for teachers and students of West Virginia's ESEA Title II, Part D school-based teacher trainer initiative. The study will employ an experimental research design and make use of technology-based desktop meters and random interval data collection pop-up screens to document the use, time, topic and function budgets of teachers and students. These methods will yield objective, detailed information about classroom integration of technology as an outcome of professional development and the impact of technology integration practices on student performance on West Virginia's tests of standards-based content. In its third year, data from the study will be used to explore the extent to which evaluation data is used by state policy-makers to inform decisions.
Wisconsin Department of Public Instruction
A Study of the Effectiveness of Three Models of Implementing Educational Technology
$1.6 million over three years
Online: www.dpi.state.wi.us/
The Wisconsin project will (a) identify three promising models of educational technology use in state schools from existing data; (b) implement the models using the Title II, Part D competitive grant process; and (c) evaluate the effectiveness of the models on student achievement using quasi-experimental methods to assign experimental and control groups, student portfolios, student self-reports, and standards-based knowledge assessments to measure student achievement.
1. Arkansas Education Department
Evaluation of the EAST (Environmental and Spatial Technology) Initiative; $1.8 million over three years.
The EAST Initiative involves the creation of interdisciplinary school-based technology labs that promote student intellectual growth and technology skills acquisition, as well as teacher training on facilitating student learning through service projects and teamwork. The study will involve 120 projects serving 9,000 students (55% rural, 25% suburban and 20% urban setting).
2. Iowa Education Department
Using Technology to Support the Scaling-Up of the Iowa Professional Development Model; $1.9 million over three years.
This multiagency, statewide field-research effort will focus on reading and mathematics instructional practice in the sixth, seventh and eighth grades in 150 public school districts, including 43 high-needs districts as defined by NCLB. The project will yield a research model for identifying and scaling up teacher training on best practices.
3. Maine Education Department
The Impact of Teachers' Professional Development on the Mathematics Achievement of Low-Performing Rural Students in Technology-Rich Classrooms; $1.9 million over three years.
The study will focus on seventh- and eighth-grade students in schools that serve low-income rural communities and have shown low performance in eighth-grade mathematics. The study promises to contribute to research-based knowledge of effective practices in mathematics education and technology integration, ubiquitous computing, professional development, and education in low-income rural schools.
4. N.C. Public Instruction Department
LANCET: Looking at North Carolina Educational Technology; $1.5 million over three years.
This project will study the implementation of the state's IMPACT professional development model and its effects on schools, teaching practices and student achievement. The project will develop and assess strategies for building the capacity for educators across the state to collect, analyze and use evaluation data for making decisions about technology programs, projects and practices.
5. Pennsylvania Education Department
Evaluation of Student and Parent Access Through Recycled Computers (eSPARC); $1.8 million over three years.
The eSPARC study project seeks to develop and test an evaluation model to measure the impact of educational technology initiatives. The project will randomly assign recycled computers to a sample of 400 fifth-graders and their families. The study will assess whether and how in-home computer and Internet access impact students and parents.
6. Tennessee Education Department
The Tennessee EdTech Accountability Model (TEAM); $1.7 million over three years.
The project will measure the effectiveness of an intervention to prepare school-based technology coaches to work with teachers to align technology use with the delivery of the curriculum to foster increased student achievement. The project will measure the effectiveness of the intervention in 37 schools and develop a replicable, validated evaluation protocol for use in all schools.
7. Texas Education Agency
Evaluation of the Texas Technology Immersion Pilot (eTxTIP); $1.9 million over three years.
TxTIP is a technology immersion pilot seeking to increase student achievement by providing each student with a wireless mobile computing device, software, online and other learning resources. The TxTIP evaluation will study the effectiveness of technology immersion in increasing middle school students' achievement in core academic subjects, technology proficiency and attendance, among other outcomes.
8. West Virginia Education Department
ED PACE: Educational Development for Planning and Conducting Evaluations; $1.4 million over three years.
ED PACE will assess student achievement in virtual foreign language courses as compared to the achievement of students in classroom-based foreign language courses. Over three years, the project will generate three scientifically based research models (a summative, formative and action research model) that can be replicated in other settings.
9. West Virginia Education Department
The Evaluation of West Virginia's Enhancing Education Through Technology Model School Project; $1.3 million over three years.
The study will make use of technology-based desktop meters and random interval data collection pop-up screens to document the use, time, topic and function budgets of teachers and students. These methods will yield information about classroom integration of technology as an outcome of professional development and the impact of technology integration practices on student performance.
10. Wisconsin Public Instruction Dept.
A Study of the Effectiveness of Three Models of Implementing Educational Technology; $1.6 million over three years.
The Wisconsin project will (a) identify three promising models of educational technology use in state schools from existing data; (b) implement the models using the Title II, Part D competitive grant process; (c) and evaluate the effectiveness of the models on student achievement.
IES Funded Projects
IES has allocated $26,640,663 to 20 different universities and independent research organizations for various technology-based research projects as part of its mission to "improve education at all levels." The amount of the grants awarded ranges from $250,000 to $5,999,744. The grantees and a summary of their projects follow.
1. CAST Inc.
Reading to learn: Investigating general and domain-specific supports in a technology-rich environment with diverse readers learning from informational text.
Project focus: Develop a computer-based instructional approach that will support readers at risk for literacy difficulties and will accelerate their development of reading comprehension, especially for informational text.
2. University of Colorado
Research on and with novel educational technologies for comprehension.
Project focus: Develop a series of computer-managed instructional activities that will help middle school, high school and college students with limited vocabularies acquire substantially larger vocabularies necessary for high-level comprehension.
3. University of Memphis
Coh-metrix: Automated cohesion and coherence scores to predict text readability and facilitate comprehension.
Project focus: Develop two automated tools to enable writers, editors and educators to more accurately estimate the appropriateness of a text for their audience and to pinpoint specific problems with the text.
4. The Pennsylvania State University
Intelligent Tutoring Using the Structure Strategy to Improve Reading Comprehension of Middle School Students.
Project focus: Address students' failure to identify main ideas from expository text, as well as give cohesive and complete accounts of what they read through a Web-based intelligent tutoring intervention for middle school students.
5. Carnegie Mellon University
Reader-Specific Lexical Practice for Improved Reading Comprehension.
Project focus: Using recent improvements in computer science, develop a search engine tailored for selecting text passages that meet very detailed student information needs, including topic, difficulty level and desired vocabulary patterns.
6. Northern Illinois University
Improving students' comprehension and construction of arguments.
Project focus: Develop and evaluate the effectiveness of an instructional program for teaching high school and college students to better comprehend, evaluate and produce written arguments using a Web-based tutoring system.
7. University of California, Los Angeles
Introducing desirable difficulties for educational applications in science.
Project focus: Determine whether interventions that slow the rate of learning and appear to enhance long-term information retention can generalize to other contexts involving middle school and college students.
8. Columbia University
Study Enhancement Based on Principles of Cognitive Science.
Project focus: Improve academic performance by changing the way children approach studying by using a computer-based study program based on principles of cognitive science, and designed to target and improve memory and learning.
9. Washington University
Test-Enhanced Learning.
Project focus: Provide a new approach to learning, memory and comprehension of written material by examining the effectiveness of using testing as a learning tool in a Web-based university course.
10. Carnegie Mellon University
The Neural Markers of Effective Learning.
Project focus: Use brain imaging to improve the design of a computer-based instructional approach to teaching key algebraic concepts. Effectiveness will be evaluated by assessing student gains on solving word problems.
11. University of Maryland, College Park
Computer-Assisted Instruction for Learning and Long-Term Retention Based on Recent Cognitive and Metacognitive Findings.
Project focus: Improve computer-assisted instruction designed to facilitate the learning and long-term retention of second-language vocabulary by creating an individualized computer tutor for children and adults.
12. LessonLab Inc.
Improving Achievement by Maintaining the Learning Potential of Rich Mathematics Problems: An Experimental Study of a Video- and Internet-Based Professional Development Program.
Project focus: Create a video- and Internet-based professional development program in which middle school pre-algebra teachers learn to identify, design and incorporate mathematically rich problems in ways that retain rich learning opportunities for students.
13. University of Texas Health Center
Scaling Up a Language and Literacy Development Program at the PreK Level.
Project focus: Examine the effectiveness of an online early literacy professional development model for early childhood teachers. The university's project promises to contribute to our knowledge of technologically supported teacher professional development.
14. Univ. of Texas Health Science Center
Scaling Up an Assessment-Driven Intervention Using the Internet and Handheld Computers.
Project focus: Investigate the effectiveness of an intervention that uses the Internet and handheld computers to help teachers link results from reading inventories to instructional practice to produce improvements in student achievement.
15. Southern Methodist University
Scaling Up Effective Intervention for Preventing Reading Difficulties in Young Children.
Project focus: Study the relative effectiveness of approaches to teaching reading when early literacy intervention teachers are provided with differing models of professional development that use computer-based systems for content delivery and coaching.
16. University of California at Berkeley
A Longitudinal Study of the Effectiveness of a PreK Multisensory Literacy Curriculum, Teacher Experience and Professional Development.
Project focus: Examine the efficacy of an interactive, multisensory technology-based literacy curriculum with at-risk preschool children, which focuses on literacy skills that are essential to reading success in the first grade.
17. The Media Group of Connecticut Inc.
Remarkable Reading Machine: A Video/Electronic Media Training Program of Evidence-Based Interventions to Strengthen Emergent Literacy Skills of At-Risk Children from Low-Income Families in Any Child Care Setting.
Project focus: Create a highly motivational video/electronic media-training program on emergent literacy interventions for at-risk preschoolers and their parents and/or caregivers that can be used in any child care setting.
18. USteach Inc.
Synchronized Multimedia E-Book Development for Reading Fluency and Comprehension.
Project focus: Demonstrate improvement in second- and third-graders' reading fluency and comprehension through sustained interaction with e-books. Their reading performance will be compared to students without computer support or with another computer-based program.
19. Quantum Simulations Inc.
Phase II: Artificial Intelligence Software for Student Assessment in Chemistry.
Project focus: Advance the state of the art in chemistry education software through the development of meaningful interactive tutoring and assessment capabilities for chemistry problem solving.
20. The NeuronFarm LLC
Training the Tutors: Literacy e-courses.
Project focus: Develop a set of eight e-courses based on empirically verified principles of learning and effective literacy instruction to improve tutor training.
John Bailey is the former director of the Office of Educational Technology at the U.S. Department of Education.
Therese Mageau is guest editor for T.H.E. Journal's "A Closer Look at SBR" series.
SETDA's Evaluation Grant Consortia
The State Educational Technology Directors Association (SETDA) has partnered with nine of the Evaluating State Education Technology Programs (ESETP) grantees to provide dissemination and networking services through their Technical Assistance Partnership (TAP) Program.
"This program is an example of the emerging trend of states looking more aggressively at partnerships with the corporate community and departments within their agency, as well as pooling their resources among states to provide stronger and more effective programs," says Melinda George, SETDA Executive Director. By teaming up, the TAP program encourages collaboration during the program development phase and provides states with a powerful, high-quality distribution channel to communicate interim and final progress about each of the grants.
Throughout the grants' three year duration, SETDA will:
1. Provide networking and collaboration tools and opportunities for TAP participants.
2. Provide interim information and success stories regarding scientifically based research via a Web site and presentations at national conferences.
3. Develop a comprehensive handbook outlining participants' work throughout the course of the grant process.
Later this month, SETDA will launch a Web site to highlight the methodologies, strategies and interventions addressed by the evaluation grants. At the conclusion of the three-year grant period, SETDA will develop a handbook, which will include a section from each of the TAP members to highlight results, methodologies and best practices.
This article originally appeared in the 05/01/2004 issue of THE Journal.