Guide Offers 5 Steps for Measuring the Impact of Blended Learning

• By Dian Schaffhauser
• 09/08/15

An organization intent on implementing blended learning into the classroom has released a guide to help school leaders figure out how to measure the effectiveness of their digital learning efforts. The 12-page "District Guide to Blended Learning Measurement" from The Learning Accelerator (TLA) provides a five-step approach to the process. TLA describes blended learning as the "effective use" of technology to achieve personalized learning and a competency-based progression of student learning.

Step 1 in the guide is to understand the similarities and differences between research and evaluation, and how each can be used to address the questions a school system wants to answer.

Step 2 examines when to measure. As Saro Mohammed, a partner at TLA and author of the guide, wrote, "It is never too early to start thinking about and doing measurement, especially if your measurement approach is one of continuous improvement." Each phase where change is expected to occur, she explained, will have its own form of measurement. For example, it might make sense to start tracking outputs in the wake of activities such as professional development. These quantitative measures can include how many teachers have been trained or how many students have experienced blended learning.

John Watson, founder of the advisory firm Evergreen Education Group, commented in a blog post that "measuring inputs and activities will help the district determine if the blended learning program was implemented with fidelity to the plan."

Step 3 discusses what to measure. While an obvious answer might be “student academic achievement,” wrote Mohammed, there are plenty of other measurements to consider, such as whatever ties back to the reason a district has implemented blended learning in the first place.

Step 4 addresses whom to measure. While students are an obvious choice, the guide stated, there are also teachers, administrators, families and community members who are stakeholders in the digital transformation. There are also "comparison groups," the non-participants who provide a benchmark against which the school or district will know for certain whether the blended learning initiative is responsible for the results being measured among the participants.

Step 5 covers how to measure, with an emphasis on reliability and validity. Reliability, the report explained, "tells us if a measure is measuring something consistently." Validity "tells us if the thing that the measure is measuring is what we think it is." Without those two components, said Mohammed, the data being generated will be "either inaccurate/noisy, misleading or both."

"Measure what you are doing, in detail. Compare what you are doing to: what you intended to do; what was done before; and/or what is typically done," the guide advised. "This ensures that there are specific practices to which your results can be linked, whether positive or negative. If there's no tracking of what is being implemented, then there's no way to know if it is or isn't working."

If the results are negative, does that mean that blended learning has failed? Not necessarily, said Mohammed. "Outcomes that don't meet expectations could mean a variety of things, and a well-developed measurement system will also tell you why your results are not as expected. This is why it is important to think through all of the steps we include in the guide, so the results you get from measuring your implementation are actionable and useful."