Evaluating the Success of Your Ed Tech Program

Jeff Mao, who helped lead the nation's first statewide 1-to-1 learning program, offers advice on how to begin an ed tech evaluation.

• By Jeff Mao
• 07/12/17

I've been a part of the education technology crowd for a long time now. My first teaching job was with Brewster Academy, a small boarding school in New Hampshire. In the fall of 1993, Brewster launched its first 1-to-1 laptop program with its ninth-grade students. We had no internet. The laptop batteries lasted fewer than two hours if you were lucky, and our software tools were mostly limited to a productivity suite.

Despite the limitations of the technology back then, I learned a lot from my time at Brewster Academy. One of the things that we did well, that I still recommend to schools today, is to be targeted and intentional about how you use the technology. At Brewster, we measured success one skill at a time and one student at a time.

A decade later, I joined the team at the Maine Department of Education that led the Maine Learning Technology Initiative (MLTI) — a much larger initiative. Whereas Brewster's total student body was fewer than 350 students, MLTI served about 35,000 students in more than 230 schools.

How did we measure success? At the state-level, we looked at metrics different from the ones at Brewster. We looked at data points that we believed were indicators of positive movement forward, such as the percentage of teachers who engaged kids with the technology in either instruction or a learning activity, who used the technology to conduct research for lesson plans, or who used the technology to aid with assessing student progress. Looking at this data, we saw positive indicators of success.

However, the question remained: Were student performance and learning improving? This question is hard to answer from the state-level and what I've learned over the years is to return to what I found at Brewster. Schools need to be targeted and intentional with their use of technology, as well as mindful of student growth at the individual skill level.

Technology's Impact in One Classroom

In 2009, MLTI published a research brief about a single science teacher in a small coastal town. Kevin Crafts taught eighth-grade science at the Bristol Consolidated School (where he continues today). Working with our researchers from the University of Southern Maine (USM), we examined student learning on a tricky science concept: the axis angle of the Earth and its impact. Crafts had two sections of eighth-grade science, and he taught the concept to the classes. He used the MLTI laptops with one group to create animated podcasts while the other group completed a more traditional poster assignment to demonstrate what they'd learned.

All students completed pre- and post-assessments, as well as a retention assessment about a month later to compare growth and learning between the groups over time. The results showed that the group that used the technology to create animated podcasts outperformed their peers and a month later they showed very good retention of the concepts as compared to the other group.

Reading through the research brief, you find other interesting details the researchers captured through their observations. They also conducted interviews that uncovered more insight into why the podcast assignment led to better student learning. I know that almost all schools and teachers will not be able to go into this much depth and, in particular, may only be able to conclude anecdotal and intuitive understanding of the "why." But with some preparation and planning, teachers can measure whether one method or another works better. Kevin Crafts no longer assigns the poster project and that's really what matters. He's found a better way.

How to Begin Your Ed Tech Evaluation

While it may seem daunting to a classroom teacher to replicate the work that Crafts and the USM researchers performed, there is a tool that could help you. The United States Department of Education commissioned the creation of an online tool — the Ed Tech RCE Coach — that not only helps you identify what you're really trying to measure but also will help you crunch the data and analyze it.

Using the Ed Tech RCE Coach requires some work and planning and I don't believe it’s necessary for each and every lesson that you teach. However, if you're considering adopting a new technology tool for your district, school, or classroom or a significant change in how you teach a skill or concept, taking the extra time to leverage the tool can help you better gauge if you're moving in the right direction.

Regardless of whether or not you use the tool, these three simple steps embedded in it can be terrific as guideposts for your teaching:

1. Identify your desired outcome;
2. Identify what success looks like; and
3. Identify the tools and pedagogical approach that you're using.

By following these simple steps, even without instituting a pre- and post-assessment or control and intervention groups, you can begin to better understand success. With these in place, you can make more direct comparisons between group performance and growth. Lacking these, you can compare this year's students to last year's students. It isn't perfect, but it's a start.

To learn more about the Ed Tech RCE Coach, you can view an archived webinar hosted by the U.S. Department of Education.

About the Author

Jeff Mao joined Common Sense Education in the fall of 2014. He provides strategic consulting to states and districts on the effective implementation of digital learning programs. Prior to joining Common Sense Education, Jeff served as the Learning Technology Policy Director for the Maine Department of Education. In that role, he provided leadership and education technology policy support to four Commissioners of Education, and he was directly responsible for the Maine Learning Technology Initiative (MLTI). He oversaw all aspects of the program including vision, implementation, policy, professional development, procurement and contract management. Jeff has keynoted international events sponsored by UNESCO, World Bank, InterAmerica-Development Bank, and the Korean Education and Research Information Service. He served on the Board of Directors of the State Educational Technology Directors Association for six years including two as Chair of the Board. Common Sense Media named him Educator of the year in 2012, and SETDA named him Leader of the Year in 2013.

Common Sense Education helps educators find the best edtech tools, learn best practices for teaching with tech, and equip students with the skills they need to use technology safely and responsibly. Go to Common Sense Education for free resources including full reviews of digital tools, ready-made lesson plans, videos, webinars, and more.