Using artificial intelligence to measure the effectiveness of professional learning

Editor’s note: This essay is an entry in Fordham’s 2023 Wonkathon, which asked contributors to answer this question: “How can we harness the power but mitigate the risks of artificial intelligence in our schools?” Learn more.

Raise your hand if you have ever sat through shoddy professional learning... Now, I bet every educator reading this would raise their hand in agreement. Here is an alarming statistic: We spend nearly $18 billion a year on professional learning in this country, yet according to Rivet Education’s research of the professional learning marketplace, only one in three teachers find the professional learning they receive to be useful.^[1] That means nearly $11 billion spent on professional learning is wasted; this is equivalent to giving every teacher in the U.S. a stipend of $3,000. Let that sink in for a moment.

And we get it. Planning and executing high-quality professional learning is hard work. Whether provided internally by school or district leaders or externally by a professional learning organization—professional learning takes time and expertise to make it effective. Similarly, teachers who create their lesson plans from scratch hope that what they teach their students is aligned with grade-level standards. Education leaders who plan professional learning also have to cross their fingers in hopes that the training they develop will actually improve teacher practice and student outcomes.

What can further complicate matters is that, as a whole, the field of education lacks reliable measures to assess the quality and impact of professional learning. In October, the Research Partnership for Professional Learning (RPPL) at the Annenberg Institute released its latest white paper. It found that “professional learning practitioners agree the field lacks the tools needed to evaluate the implementation and outcomes of professional learning programs effectively.”

At Rivet Education, we know all too well the importance of evaluating the quality of professional learning. Rivet Education’s Framework for Curriculum-Based Professional Learning defines for educators the characteristics, types, and structures that construct high-quality, curriculum-based professional learning (CBPL). Our flagship tool, the Professional Learning Partner Guide (PLPG), is the only evaluation process in the country that assesses the quality of CBPL services. While the PLPG serves as the industry standard for evaluating CBPL, the evaluation process is time-consuming (upwards of 500 hours per review cycle) and currently only reviews professional learning materials, not the live delivery of the training.

But what if we can harness the power of artificial intelligence to increase the efficiency and effectiveness of measuring the quality of professional learning?

Using AI to measure professional learning quality in school districts

Imagine you are a math supervisor in a mid-size school district and your Chief Academic Officer has tasked you with creating a two-day workshop to launch Illustrative Mathematics (IM) with teachers in grades 6–8 in your district. This is the only training teachers will receive until October when your district typically hosts a professional development day, and the central office couldn’t afford to have IM come in to provide the training themselves. The problem is that you are also new to IM. Your only experience up to this point with a comprehensive math curriculum was with one that was not high quality nor aligned with your state’s standards. So, you design the training, hoping you have done enough to adequately prepare your teachers to use the new curriculum on the first day of school.

A generative AI application could be used in this scenario to review the scope and sequence of the workshop and provide the math supervisor with feedback on how their training will prepare teachers to teach IM successfully. Here is an example of an AI prompt the math supervisor could use to eliminate some of the guesswork of planning effective professional learning:

Here are the training materials for a two-day workshop designed to help teachers understand the design principles and arc of learning of Illustrative Mathematics. Suggest three ways I could improve them to help teachers new to the curriculum use it more effectively.

Using AI to measure the quality of professional learning organizations

Professional learning organizations and those who evaluate their services, like Rivet, can also use AI to evaluate the quality of their PL. Consider Rivet’s evaluation process for the Professional Learning Partner Guide (PLPG). This involves teams of expert reviewers evaluating applicants’ materials, such as presentations, handouts, guidebooks, course syllabi, coaching notes, etc. These are compared alongside Rivet’s criteria to assess if the organization provides significant evidence of robust, HQIM-aligned professional learning services. Professional learning partners that meet these criteria are profiled in the PLPG. While we are confident that our process holds a high bar for quality services, we also recognize that factors such as the quality of the facilitator in the room impact PL effectiveness.

As an organization deeply valuing quality, we asked ourselves: What if AI could review and rate the quality of in-person professional learning and reduce the human capital burden needed to observe this training in person? This would require Rivet to feed AI a lot of information to replicate the knowledge and experience of our expert review team, but it is an idea worth pursuing. Here is an example of what it could look like:

Here is the transcript from a live professional learning workshop from Pelican PL Services on internalizing unit 2 of EL Education. Can you compare the transcript to the professional learning materials submitted by Pelican PL services for this training to determine if the facilitator delivered the session as intended? Then, can you compare the transcript to the Ongoing for Teachers-specific Indicators for High-Quality Professional Learning in the Professional Learning Partner Guide’s scoring and evidence guide to determine if the workshop would receive a passing score of 7 out of 10 points, meeting Rivet’s definition of high-quality professional learning?

While both prompts provide examples of how AI can be leveraged to determine the quality of professional learning, the full extent of AI’s role in K–12 education is uncertain. With AI’s potential comes both excitement and trepidation. At Rivet Education, we think AI can play a valuable role in filling in gaps in professional learning that currently exist in the market. We have a tool at our fingertips that can eliminate the guesswork and uncertainty in planning and delivering professional learning, quickly and succinctly measure its effectiveness, and foster growth and innovation in schools nationwide. It is up to us to unlock its potential.