For years, millions of U.S. students have taken the NWEA MAP Growth assessment. Data from these computer-adaptive assessments—which cover math, reading, language usage, and science—can help teachers determine which students need remediation or other supports and in which topic areas. This has been particularly important in the wake of Covid education disruptions beginning in spring 2020, which sent achievement levels plummeting; the need for supports to remediate that epic learning loss continues today. A new analysis, conducted by the organizations involved in its implementation, looks at an ambitious effort to boost MAP achievement.
Starting in 2018, NWEA teamed up with online education provider Khan Academy to develop a tool to help increase student achievement on MAP Growth’s mathematics assessments. Called MAP Accelerator, it uses previous achievement data to create supplemental content customized for each student. It is geared toward helping students build skills up to mastery level using a combination of video instruction and intensive practice lessons with detailed performance feedback, gradually moving students on to higher and harder content.
MAP Accelerator was rolled out to NWEA-participating schools in fall 2020, coincidentally just when evidence of Covid-related learning loss was beginning to surface. Ninety-nine districts participated. Analysts reviewed data from more than 181,000 students in grades three through eight who utilized the tool in 2020–21. There was a roughly even split between males and females; 34 percent were White, 12 percent were Black, 5 percent were Asian, and a noteworthy 35 percent were Hispanic. The authors make clear that this is not a nationally-representative sample and wasn’t intended to be, as the MAP Accelerator is marketed as a support for traditionally-underserved students. Fifty-two percent of students in the study sample attended schools where the majority of students are eligible for free or reduced-price lunch.
The analysts use a quasi-experimental, pre-test post-test design. Random assignment was not possible, so analysts instead broke students into four groups: no usage, less than fifteen minutes of usage per week, fifteen to thirty minutes of usage per week, and more than thirty minutes of usage per week. They utilize “judicious” statistical controls to help mitigate confounding variables and look at outcomes of other non-math MAP Growth assessments to attempt to identify and control for selection bias.
Only 5 percent of students used MAP Accelerator at the recommended dosage of thirty or more minutes per week. Most students (45 percent) fell into the less than fifteen minutes per week category. A whopping 41 percent did not use the tool at all. Students in the highest-usage group spent, on average, about three twenty-minute sessions per week over twenty-four weeks. While all groups showed growth in test scores from fall 2020 to spring 2021 (based on whatever their pandemic-impacted starting point was), the highest-usage group registered growth that was 0.26 standard deviations higher on average than similar students who used the platform for less than fifteen minutes per week. This general trend was consistently observed regardless of student race and ethnicity, gender, and school eligibility for free or reduced-price lunch.
While there was little evidence of self-selection bias, some evidence of other confounding variables was detected. Adjusting for these pointed to a large and statistically significant boost in math achievement based directly on the amount of accelerator usage (e.g., practice on geometry led directly to an increase in achievement on the geometry assessment). Impacts were significantly lower for students hailing from districts with greater than 20 percent English language learners. The analysts note that findings for poverty level and English learner status were at the school level rather than at the student level, rendering them less accurate.
We must be careful not to read more into these data than is here. First and foremost, this is not a randomized control trial. But equally important, would even the non-causal outcomes observed have looked the same in a normal school year? All of the students involved had been impacted by the pivot to remote learning in spring 2020, and most had seen their math achievement plummet as a result. We have no data on which students in this study had returned to in-person learning by the fall or spring, nor which students had access to broadband internet and connected devices.
All we can safely conclude is that concentrated practice with the MAP Accelerator appeared to increase achievement on MAP math assessments beyond what would have been expected without their use. But this is good enough news for now. Heaven knows we’re very clear about the problem; we need more data on possible solutions, and this one looks very promising so far.
SOURCE: Kodi Weatherholtz, Phillip Grimaldi, and Kelli Millwood Hill, “Use of MAP Accelerator associated with better-than-projected gains in MAP Growth scores,” Khan Academy and NWEA (August 2022).
