COVID-19 and Learning Loss
By Keith Devlin @profkeithdevlin
How do people learn? After being in the learning and education world, one way or another, all my career, I don’t have a good answer to that question. Even when I apply it to myself as a learner. This is, in part, why I used the more cumbersome phrase “learning and education world” just then. I don’t understand, in anything beyond a superficial way, the relationship between learning and education.
I don’t have much problem with what learning is. Sure, I don’t really understand how it happens. The familiar story about building new neural pathways and strengthening existing ones seems plausible as far as it goes, though it ultimately leaves me unsatisfied; but as a model to guide action, it seems fine. The “how it happens” mystery aside, however, it seems pretty clear that natural selection tends to favor agents that can learn from their experiences, i.e., adapt their behavior accordingly, to their future advantage. Indeed, at least for we humans, we seem to be hard-wired to get pleasure from learning that we perceive as valuable to us. In short, learning is both a necessary and a good thing, I’m all for it, and I have always had an urge to help others to learn, both as a parent and professionally as an “educator.”
If all that society meant by the word “educator” was helping others to learn, I would not have put the word in quotes at the end of that last paragraph just now. But as we all know, it means a lot more than that. In particular, it has a systemic meaning. Societies have education departments, ministers and secretaries of education, education policies, and the like. Education is a product to be sold and bought. There are education resources and education technologies. And there is education content.
It’s when the word “content” creeps in that I have a lot of trouble with the relationship between education and learning. In its most simple (and widespread) form, talk of “content” views education as a process of pouring buckets full of “educational stuff” into the partially-full heads of students. And then, in the name of system accountability, measuring how much of that content did not somehow spill or leak away overnight.
It’s not that talk of “educational content” does not have some value. But it is just one of several ways of looking at one part of what is involved in helping people to learn things that may benefit them and on which society places value. At least just as significant, and to my mind far more so, is that education is a process not of filling a bucket but of lighting a fire—creating in people a desire to learn and helping them acquire the skills for learning: reflection, asking questions, seeing things from different angles and through the eyes of others, and being open to change in the face of new evidence, information, or experiences.
“Education is not filling a bucket, but lighting a fire.” This quotation from the Irish intellectual William Butler Yeats (1865-1939) has achieved meme status, and can be found all over the Web.
As someone who gets paid to be an “educator,” my uncertainties as to what it is that I actually deliver, or why exactly people and organizations say I am good (and occasionally bad) at it, as they do, have not stopped me doing it. Those uncertainties have, however, left me wondering if the most important ingredient I bring to my “educator” life is simply wanting to help others learn. (I mean really wanting to.) All else, I suspect, may be little more than interpersonal social behavior driven by that inner desire—or maybe I should say inner need—perhaps wrapped up in some teaching skills and tricks I’ve picked up over the years.
At any rate, that’s the background I brought to reading a recent report from PACE (Policy Analysis for California Education, an independent, non-partisan, universities-directed research center in California) titled “COVID-19 and the Educational Equity Crisis.”
The paper’s focus is on the potential (and, it seems, real and highly significant) inequitable impact of the pandemic on student learning. As the authors say early on:
“The pandemic has exacerbated a preexisting equity crisis in our state. Due to inequitable access to healthcare, income inequality, and disproportionate employment in high-risk, ‘essential’ jobs, low income, Black, and Latinx communities are suffering most from the health and economic impacts of the pandemic. Most California schools have been physically closed since Spring 2020, and students in these groups have been doubly disadvantaged by distance learning. Low-income and rural students may not have reliable access to computers or Wi-Fi, making it difficult to access online instruction. Others do not have appropriate conditions for learning at home and struggle with food or housing insecurity or with limited adult support. Distance learning is harder to implement for students in certain groups, including students in the youngest grades, students with disabilities, and students learning English. While some analyses to date have shown inequity in grades, attendance, and enrollment, our analysis is the first to use student-level demographic data to show differences in learning during the pandemic in California by income and language status.”
What brought me up short was the use of the term “learning loss,” and in particular the following graph.
Image from the PACE report discussed in this essay
True, the graph is a theoretical one, drawn to illustrate a concept and establish the framework and the vocabulary for the remainder of the report. But what exactly are we to take “learning” as meaning in this context, and in what sense has it been “lost”?
I’m by no means the first mathematician or mathematics educator to be triggered by the now common use of the term “learning loss.” For instance, my long-time friend and colleague John Ewing, currently President of Math for America, expressed a similar concern in a recent opinion piece written for Forbes, titled “The Ridiculousness of Learning Loss.” In his case, the trigger was an earlier report on COVID-19 and “learning loss” by McKinsey and Co.
Like me, Ewing is not trying to dismiss the study, or to minimize the potential fallout to society caused by the pandemic, particularly the present generation of school-aged children, and even more particularly children from socioeconomically disadvantaged backgrounds. Our point—at least, it’s my point, and I take it to be Ewing’s as well—is that use of the term “learning loss” and iconic graphs like the one above significantly oversimplify a complex issue that none of us really understands.
The problem is not that the authors of those reports, or indeed any experts in the education world, view the situation in such a simple way. The PACE authors go to great pains to try to state clearly what the issues are, and are completely open about the complexities of the issues and the limitations of their study. What worries me (and I suspect Ewing) is that when reports written by and for experts are picked up by the mainstream media, and then spread out organically though society at large, what began as a way to present study findings in a clear way to experts, can convey the impression that we (the experts) know and understand way more than we do, and in worse case scenarios can lead parents, mainstream teachers, and government officials to reach unfounded conclusions or make disastrous decisions. [It’s probably relevant here to note that Ewing and I are both professional mathematicians who have spent large parts of our careers on public outreach activities, including mathematics education advocacy. We are sensitive to how easy it is for well-meaning messages to do wrong.]
Ewing’s article does a great job of summarizing the particular dangers of using the term “learning loss” in relation to the pandemic, so I am not going to repeat his points here. You should definitely read his article; it’s short and to the point.
I will just point to one finding in the PACE report that indicates just how much complexity there is in the issues reported, and how totally misleading that original “learning loss” graph can be if taken to represent anything real. The figure shown below compares student performance on two standardized assessments, in 2020 (when teaching was done remotely due to the pandemic) and the previous year (when schools were operating normally).
Figure taken from the PACE report discussed in this essay. MAP and STAR are standardized tests. The green “learning gain” bars are surprising and cry out for explanation.
Look at the entries for math. While students in grades 4 through 7 did worse in 2020 (the orange bars represent “learning loss”), those in grades 8, 9, and 10 actually did better (the green bars show “learning gain”). A second chart, included in the paper (Figure 3) but not here, shows that this phenomenon was observed both for socioeconomically disadvantaged (SED) students and for non-SED students (when the data were separated). So the surprising results are probably not a result of access to technology or parental assistance in the home. But then, what is going on? I don’t know, and neither do the PACE authors, as they freely admit. (Good research usually raises at least as many questions as it answers.)
What, if anything, these charts tell us about student learning, I have no idea. As the PACE authors observe, what they are actually measuring is not learning (whatever that is), but student performance on standardized tests. In other words, volume of gas in the automobile tank rather than the power of the engine.
As with our automobiles, maybe so with our children in schools. For most of us, our cars remained in the garage for much of last year. Some of the gas in the tank probably evaporated, and likely we did not fill up the tank anything like on a regular basis. But if you were to go out today and find the meter showing the tank is only a quarter full, you surely will not conclude your car has less power than a year earlier when it was full and you were driving daily. Sure, you’d better check the oil, and maybe take it easy the first time you drive it, if you really have not used it much of late, and maybe even get a service. But it might not take too much time and effort to get it back up to speed.
Does that analogy really work for kids who have been educated at home all year? I have no idea. But it might. And that “might” is important. There may be no major long term damage done to student learning due to the pandemic. It may be a transient thing. (Personally, if my children were still of school age, I would be much more worried about possible long term effects on psychological development and sociability. But I know even less about that than I do about mathematics learning!)
The one thing that does come across clearly from the PACE study, however, is that there have been negative effects. Specifically, the effects of the pandemic on today’s students, whatever they are, do seem to be far greater for SED students. In other words, the pandemic seems to have led to an increase in inequity. The authors end with these words:
“Finally, analyzing whether and how student learning in ELA and Math has been affected is only one piece of understanding what students have experienced during the pandemic and how school will need to be reorganized to meet their needs. The pandemic has introduced a great deal of hardship into many students’ lives, and leaders at the school, district, and state levels will need tools to assess students’ social-emotional well-being, including physical and mental health needs, social systems of support, and general readiness to learn. Ultimately, addressing students’ learning loss will require a student-centered approach that puts family and student relationships first, and a systemic transformation in how schools address the overlapping learning, behavioral, and emotional needs that support effective learning and teaching.”
There is a powerful message there. “Learning loss” is an unnecessary distraction. If I were granted a friendly edit, I’d suggest the authors delete the word “learning” in their final sentence and just refer to “student loss.” The rest of the statement is then fine. Which means things in the world of education and learning are not fine.