Student Academic Recovery: Which Strategies Are Most Effective?

• By David Nagel
• 10/11/23

Setbacks in student achievement resulting from school shutdowns between 2020 and 2022 promise to be a pressing issue for some time to come. The good news: Research shows some effective ways to accelerate academic recovery for those students. The bad news: Not all schools are implementing these approaches, and some are implementing programs that have no positive impact — as well as programs that can lead to even greater setbacks.

The research and assessment organization NWEA, which became a part of HMH earlier this year, recently created a report, "Accelerating Student Academic Recovery," spotlighting the decline in student achievement resulting from pandemic-era policies and the evidence for success for various approaches to alleviating the problem.

As the report, created in conjunction with The Annenberg Institute at Brown University, noted: "Schools have always looked for ways to accelerate the progress of students achieving below grade-level. However, researchers have found that some of the most common approaches to helping those students catch up to their peers (such as grade retention, accelerating entry into advanced courses, and providing optional, on-demand virtual resources) tend to be ineffective. At times, such interventions even exacerbate inequalities, having negative long-term effects that outweigh any benefits students’ may experience in the short term."

We had a chance to get insights into the research from Ayesha K. Hashim, one of the co-authors of the report and a research scientist for NWEA.

THE Journal: What is your biggest takeaway from this research?

Ayesha K. Hashim: Students remain far behind from pre-pandemic levels of achievement. School districts that serve higher percentages of low-income and minority students, and that offered remote or hybrid instruction for longer periods of time during the pandemic, have also demonstrated some of the largest setbacks in learning from the pandemic with limited signs of recovery. Clearly, the problem of unfinished or delayed learning from the pandemic is a problem that is going to persist for some time, and school districts will need to engage with evidence-based strategies to help accelerate student learning for the long term.

Our brief identifies interventions that have strong evidence of effectiveness, such as high-dosage tutoring, summer school programs, and double-dose math classes. Districts should implement these high-impact interventions so that students can have access to meaningful opportunities to catch up on unfinished learning, especially for students in district contexts and in particular grade levels (e.g., middle school) that have experienced large setbacks to student achievement. It is also important that districts maintain fidelity to design principles of these interventions to ensure program effectiveness in practice, and maintain implementation approaches that can ensure program success. These implementation approaches include maintaining clear and transparent communication with parents, providing scaffolding for grade-level content, and fostering social and emotional connections with students.

We also identify interventions that have more mixed or insufficient evidence of effectiveness and explain why. Interventions with mixed evidence of success include academically focused after-school programs, computer-assisted learning, and increasing instructional time by extending the school day or year. While districts may want to consider these intervention approaches, we caution that there are limited causal studies demonstrating program effectiveness and limited information on design principles that can optimize student learning.

Finally, we caution that districts should avoid adopting interventions with insufficient evidence of effectiveness, such as grade retention, interventions that replace core instruction, and interventions to accelerate learning at scale. We have little evidence that these have been effective at raising test scores as they have been implemented to date. Prior studies have also raised important equity concerns with regard to how these interventions are implemented in practice. It is possible that these strategies might have other benefits for students that research has not been able to observe or study to date.

THE Journal: How is technology involved in effective programs aimed at improving student achievement?

Hashim: Our brief identifies high-dosage tutoring, summer school, and double-dose math classes as effective interventions for recovering student learning. In general, there is limited evidence on the role that technology can play in facilitating these interventions. For example, in the case of high-dosage tutoring, there is some evidence to suggest that virtual or blended tutoring can be effective (see Carlana & La Ferrara, 2021 or Pellegrini et al., 2018). However, the most rigorous evidence on the efficacy of tutoring programs comes from evaluations of in-person tutoring programs (e.g., Nickow et al., 2020). Therefore, whether virtual or blended tutoring programs can be as effective as in-person programs remains an open question.

Nonetheless, there are some consistent design principles across high-dosage tutoring, summer school, and double-dose math classes that could be facilitated by technology. To be effective, these programs require that students have access to structured lesson plans, high-quality instructional materials, personalized instruction and feedback, and that students participate in regular formative assessments to track student learning progress. Districts and schools should use technology to the extent that such tools facilitate these kinds of high-impact learning conditions. The one caveat being that, even with the use of technology, instruction in these programs is still primarily driven by the tutor and not the technology.

THE Journal: Again in terms of technology, what are the biggest mistakes schools make in the application of technology for intervention/remediation?

Hashim: Our brief does not focus on how schools use technology as part of interventions. That said, we do highlight interventions that have mixed evidence of success that make use of technology, including computer-assisted learning programs (e.g., ALEKS, Zearn). There is some evidence that the programs can boost math scores when they are used to supplement instruction. However, implementation and programs vary widely, and we have little evidence on how often or how long is optimal for students to use these programs, what spending time on these programs should replace in terms of students’ schedules, which students are likely to benefit most from them, etc.

Finally, our brief also highlights some of the limited effectiveness and unintended consequences (inequitable participation) that can result from optional, on-demand tutoring programs. Here’s the bullet from the brief:

• Optional, on-demand tutoring programs are common but unlikely to be used by targeted students as intended. Recent evidence on a California charter district’s use of an on-demand, virtual math tutoring program found that, without nudges to participate, only 19% of middle and high school students ever accessed the platform. Struggling students were even less likely to log on, raising concerns that opt-in resources may exacerbate inequalities rather than reduce them.

THE Journal: How widespread are the methods that this research shows to be ineffective? And how does the message reach education leadership so that they stop focusing their efforts on these methods?

Hashim: Our brief identifies grade retention, interventions that supplant (rather than supplement) core instructional time, and interventions that expose students to accelerated learning content as ineffective strategies. We have little evidence that these have been effective at raising test scores as they have been implemented to date. It is possible that these strategies might have other benefits for students that we have not been able to measure or study to date.

We do not have data on how many districts and schools are currently using strategies such as grade retention, supplanting core instruction, or exposure to accelerated learning content to recover student learning from the pandemic. That said, many of these strategies were commonly used by states and districts prior to the COVID-19 pandemic and could therefore still be used in the present.

• As of 2019, 18 states had mandatory retention laws requiring students to repeat a grade-level if they do not attain grade-level proficiency on state summative assessments (primarily in reading and in early grade levels, see Modan, 2019).

• Much of the evidence on interventions that supplant rather than supplement core instruction comes from programs serving students with learning disabilities. Over time, students with disabilities have benefited from increased access to general education classroom settings where they can receive core instruction, as opposed to being taught in separate classroom settings. However, gaps in access to general class settings remain. As of fall 2017, only 63% of students with disabilities spent most of the school day (80% or more of their time) inside general classes in regular schools (McFarland et al., 2019).

• Finally, increasing student access to advanced courses, especially in math, has been a national policy goal for the United States since the early 1990s. Taking eighth-grade enrollment in Algebra 1 as an example, currently over 80% of students in public schools have access to Algebra 1 as an advanced math course (U.S. Department of Education, 2018).

In terms of conveying this message to education leaders, our brief intends to raise awareness about rigorous evidence on the efficacy (or lack thereof) of intervention strategies in terms of raising student achievement. By translating evidence from rigorous studies that document the negative impacts of certain interventions on student achievement, or that highlight alarming inequities in terms of how interventions are implemented, we hope to equip district and school leaders with information that can guide their decision-making moving forward. Of course, having sustained interaction and conversations with practitioners on how to make sense of this evidence and incorporate it into policy decision-making can only be an added advantage or resource for education leaders.

THE Journal: We see that pulling students out of core instruction for interventions has a negative impact. How can interventions be delivered without cutting into core instruction?

Hashim: Scheduling time for interventions during the school day is a challenging task. District and school leaders will need to be flexible and creative with scheduling, as “one size fits all” approach may not work for all students and their distinct needs. When possible, schools should choose scheduling strategies that require the least investment of school resources and time, and that are the least disruptive to a student’s day and schedule (including their access to grade-level instruction). Some strategies that districts and schools have shared include scheduling interventions during lunch, immediately before or after school, and during transition periods or independent learning times. When more time is needed, district and schools may consider temporarily scheduling interventions during special topics (e.g., physical education and arts) or elective times, although a downside to this strategy is taking time away from immersive or well-rounded educational experiences. Another option to keep in mind is that districts and schools can intensify interventions without having to add more time. As suggested in our brief, one way to maximize intervention intensity is to adhere to design-based principles that maximize student achievement impacts (e.g., small group ratios, frequent meetings, aligning content with grade-level instruction, etc.).

THE Journal: Is there an equity component in your findings? Can you talk about it?

Hashim: As mentioned earlier, we are most concerned by the inequitable consequences of the pandemic for students in districts that serve high proportions of low-income and minority students and that offered remote or hybrid instruction for an extended period in the pandemic. We are also concerned about the setbacks to student learning for students in later grades (middle school) who are showing the slowest rates of recovery now that schools have re-opened and have the least amount of time in K–12 schools to make up for missed learning opportunities in the pandemic. It is imperative for education leaders to address these learning gaps if we want to prepare students for college and career pathways and to fully participate in our economy and democracy. Implementing evidence-based strategies to recover student learning is part of how education leaders can begin to do this important work. Yet, as we have shown in other reports, districts that serve high proportions of low-income and minority students are facing some of the greatest challenges when it comes to implementing evidence-based strategies to recover student learning at scale (see Carbonari et al., 2022). Therefore, districts will likely need additional time, funding, and resources to implement pandemic recovery strategies for the benefit of students.

References

• Carbonari, M., Davison, M., DeArmond, M., Dewey, D., Dizon-Ross, E., Goldhaber, D., Hashim, A., Kane, T., McEachin, A., Morton, E., Patterson, T., Staiger, D. (2022) The Challenges of Implementing Academic COVID Recovery Interventions: Evidence From the Road to Recovery Project. Research Report. Cambridge, MA: Center for Education Policy Research, Harvard University. Retrieved from https://cepr.harvard.edu/sites/hwpi.harvard.edu/files/cepr/files/the_challenges_of_implementing_academic_covid_recovery.pdf?m=1677190353

• Carlana, M., & La Ferrara, E. (2021). Apart but Connected: Online Tutoring and Student Outcomes during the COVID-19 Pandemic.

• McFarland J., Hussar B., Zhang J., Wang X., Wang K., Hein S., Diliberti M., Forrest Cataldi E., Barmer A., Nachazal T., Barnett M., Purcell S. (2019). The Condition of Education 2019 (NCES 2019-144). National Center for Education Statistics. https://nces.ed.gov/pubs2019/2019144.pdf

• Modan, N. (2019). 50 States of Ed Policy: Do third-grade retention policies work? https://www.educationdive.com/news/the-50-states-of-education-policy-do-3rd-graderetention-policies-work/559741/

• Nickow, A., Oreopoulos, P., & Quan, V. (2020). The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence. National Bureau of Economic Research Working Paper Series(w27476).

• Pellegrini, M., Lake, C., Neitzel, A., & Slavin, R. (2020). Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis. Manuscript submitted for publication.

• Roschelle, J., Cheng, B. H., Hodkowski, N., Neisler, J., & Haldar, L. (2020). Evaluation of an Online Tutoring Program in Elementary Mathematics. Online Submission.

• United States Department of Education. (2018). A leak in the STEM pipeline: Taking Algebra 1 early. Retrieved from https://www2.ed.gov/datastory/stem/algebra/index.html#:~:text=Overall%2C%2080%20percent%20of%208,I%20in%208th%20grade.