U.S. Dept. of Ed Tells Schools To Carry Out Research on Educational Technology: Really?

• By Cathie Norris, Elliot Soloway
• 11/28/16

In a follow-up to its #GoOpen Initiative, the U.S. Department of Education is calling on educators — initially "district-level administrators and some school-level administrators" — to carry out research studies to evaluate the effectiveness of using technology:

• "Districts and states are spending millions of dollars buying educational apps [technology applications, tools and platforms], many of which have minimal evidence supporting their effectiveness …"  –Katrina Stevens, deputy director, United States Department of Education’s Office of Educational Technology
• "Additionally, once a purchase is made, there is often no systematic process for reviewing the effectiveness of ed tech tools before renewing contracts, which collectively can run into the millions of dollars …. It is fair to say that we have vastly increased our technical capacity to enable high-quality digital learning in our schools. However, our understanding about what works in what context has not kept pace. The need to make good decisions based on evidence, as opposed to relying on marketing hype or the buzz among a small group of peers, is critical." (–also K. Stevens)

To support schools in carrying out scientifically valid, classroom-based research, the U.S. Department of Education’s Office of Educational Technology has created a website called: RCE Coach — available in beta today. The "Rapid Cycle Evaluation Coach" is a series of questions that a district is supposed to answer. Based on a district’s answers, the RCE Coach provides suggestions on what type of research study ought to be conducted and provides information on how that study should be conducted.

For example, in the "Determine Your Approach"  component of the RCE Coach, it asks:

1. Are you already using this technology?  (Our comment: not clear who "you"  is, exactly)

Yes or No — are the possible responses

2. Who are the technology users?

students, teachers, others — are the possible responses

3. Are all subjects using the technology?

Yes or No — are the possible responses

Let’s assume the educator answers:

1. Yes  (you are already using this technology)
2. Students (are the technology users)
3. Yes (all the subjects are using the technology

Then, the RCE Coach outputs this advice:

"Conclusion: Based on your answers, neither an experimental nor matched design is appropriate. You indicated that all students are using the technology. To evaluate your technology, you need to compare a group using the technology with a group not using the technology. Please reach out to us at [email protected]  and we can help you determine a research approach. In the meantime, you can still explore the site."

There is also a downloadable guide — 1 page — to provide additional information.

Stop!  Is the U.S. Department of Education’s Office of Educational Technology (hereafter abbreviated OET) really expecting that educators — school administrators and teachers, since teachers by definition are in the classrooms under investigation — will conduct scientifically valid, empirical studies of classroom use of educational technology supported by a website containing a series of questions, answers and a few PDF files?  Or will the good folks at https://mathematica-mpr.com/  individually advise (i.e., hand hold every step of the way) America’s almost 100,000 schools on how to conduct scientifically valid, empirical studies of classroom use of educational technology?

Really?

So, K–12 no longer needs trained researchers who spend several years in a graduate program in an institution of higher learning, taking multiple courses in experimental research design and statistical analysis, and being mentored in carrying out empirical studies by faculty who have spent their careers carrying out scientifically valid, classroom-based research. Just let K–12 educators do it — design, carry out, analyze and publish scientifically valid, classroom-based research!

It gets worse. OET wants schools to follow the "agile movement"  and engage in rapid cycles of empirical studies:

• "The RCE Coach can provide ongoing data more frequently … A three-month pilot period with RCE Coach will enable us to evaluate effectiveness …"

So, in addition to simply doing scientifically valid, classroom-based research, schools are expected to do rapid cycles of scientifically valid, classroom-based research.

Really?

We have seen this movie before: In its #GoOpen effort, the OET has encouraged K–12 teachers to use OER — Open Education Resources — to create curriculum to replace textbooks. Creating coherent, cohesive, standards-aligned lessons are challenging enough (please see our blogpost), but IOHO, in suggesting that teachers and administrators now conduct scientifically valid, classroom-based, empirical research, OET has crossed the line of credibility.

Yes, there is no question that scientifically valid data, not "marketing hype," are needed by schools to make better decisions concerning educational technology. But schools are not prepared, not equipped and not necessarily even inclined, to do the time-, care- and detail-demanding work that goes into gathering such data! 

What will OET suggest that educators do next? Write their own educational software? Ooops — been there, done that! Remember when teachers were going to learn to program in BASIC and create educational software for their students? That idea worked out really well.

Here’s a novel thought: if K–12 schools want to carry out scientifically valid, classroom-based, empirical research why don’t schools partner with university- and think tank-based researchers who (1) have the skills to do the research properly and (2) are hungry to partner with K–12 school districts?! Historically, university- and think tank-based researchers have partnered with K–12 schools to carry out scientifically valid, classroom-based, empirical research. Is that process so broken that the OET feels that K–12 schools should go it alone?

To be fair, the RCE Coach is beta-ware — the RCE Coach website is not complete. But, frankly, filling out the RCE Coach website won’t fix the problem. Our concern is that the very notion of K–12 educators doing scientifically valid, classroom-based research by themselves is wrong-headed.

N.B. As we are sensitive to the current political situation in the United States, we want to be clear about the scope of our comments in this blogpost:  we are being critical only of a particular policy being advocated by the U.S. Department of Education’s Office of Educational Technology.  

Bottom line: K–12 research and development, e.g., developing digital curricula, carrying out scientifically valid, classroom-based research, is a most challenging enterprise. Historically, all manner of individuals and organizations have partnered with K–12 institutions in that R&D effort. However, it appears that the OET is pushing schools to withdraw from those partnerships and go it alone. Piling more and more onto the backs of K–12 educators can’t be a strategy for effectively moving K–12 public education into the digital age. That said, perhaps we are missing something; if so, we call on the U.S. Department of Education’s Office of Educational Technology to help us better understand its new plan for educational technology research.