Episode 10

Helping Students Adopt Retrieval Practice with Robert Ariel

[MUSIC PLAYING] STEVEN ROBINOW: This is Teaching for Student Success. I'm Steven Robinow. Today my  guest is Dr. Robert Ariel, Assistant Professor of Psychology at Virginia Wesleyan University in Virginia Beach, Virginia where he is the Director of the Self-Regulation, Education, and Aging Lab. Dr. Ariel will discuss his research on a simple approach that may help students adopt a proven study method, retrieval practice, that improves long-term student learning and student success.

If you are unfamiliar with retrieval-based learning, please listen to my interview with Dr. Jeffrey Karpicke in episode 4. Helping students adopt evidence-based study practices, breaking years of experience with useless study habits is not easy, but it can be a very important piece in helping students succeed.

Dr. Ariel's research focuses on understanding and improving the metacognitive monitoring and control processes that influence self-regulated learning across the lifespan, meaning, if I understand correctly, that he uses the behaviors and thought processes we individually use to monitor our own learning and to improve our long-term learning. And he does so across the age spectrum comparing how we do or don't improve as self-regulated learners as we age. He's shaking his head, so that's a good sign.

Today we will discuss his publication entitled, "Improving Self-Regulated Learning With a Retrieval Practice Intervention," published in 2018 in the Journal of Experimental Psychology, Applied. This research was funded by the Spencer Foundation.

In this paper, Dr. Ariel tests the hypothesis that a relatively straightforward prompt can help students adopt a learning strategy in a first test and then asks whether or not the same students will utilize this learning strategy in a later, similar, but unrelated situation. Welcome, Robert. Thank you so much for joining us on Teaching for Student Success.

ROBERT ARIEL: Thank you for having me. I'm excited to be here.

STEVEN ROBINOW: I'm super excited that you're here. I love this topic. Before we get to your research, though, could you please tell us a bit about your background?

ROBERT ARIEL: Sure. Memories shape and define us. Like who you are is based on this record you have in the past. So I was just like, how do you know yourself and how do you know what you know? And that led to my interest in metacognition, which is all about knowing what you know. So it's monitoring or assessing your learning. Can people do that? When are they accurate?

And it turns out people aren't always accurate. They experience what we call metacognitive illusions where they often believe they've accurately learned something when they actually haven't. So just interested in improving learning because an educated world is a better world, so in order to improve learning, we need to understand how people learn. What are the decisions they make when studying material? Do they make optimal decisions?

And then using that knowledge and using what we know about cognitive processes to change how they study material and the change in decisions they make and help them make better decisions so that they can better retain information. So like you mentioned, this focus on understanding and improving, and that's a core aspect of my research process.

I want to understand how people learn and understand how we can change their behaviors and cognitions to improve their learning, whether that's changing their metacognition, which is, again, their ability to assess what they know. And that's important because people need to be able to accurately discriminate between what they know and what they don't know in order to make effective decisions about how to study material.

And it turns out, again, people aren't always accurate at doing this. They're often overconfident, and that's problematic because they might stop studying material before they've sufficiently learn it.

STEVEN ROBINOW: Right.

ROBERT ARIEL: Sometimes they're even under-confident and that's also problematic because they might study inefficiently, or in some cases they might not choose to engage in a task at all because they believe it's beyond their ability. As I progressed throughout my career, I became much more interested in applied basic research and taking this knowledge that we have about learning and the science of learning and actually implementing it in the classroom. How do you do that?

Because I think that we're missing that step in a lot of cases. You have cognitive psychologists in the lab that have a lot of great recommendations for how to learn material, but the question becomes, can we actually translate those into effective learning interventions? And in some cases we haven't been taking the steps to see if we can do that.

Here, Virginia Wesleyan University, we have 1,500 students. We have, I think, 48% minority enrollments, 27% of those being Black students. A quarter of our students are first-generation students, so if you're interested in reducing equity gaps, we're right on the front line for that battle. So it's a place you want to be and it's the place you want to be if you're interested in improving the pipeline in your discipline, because you have the opportunity to train the next generation of diverse scholars in your field.

STEVEN ROBINOW: So we're going to talk about this disconnect in what people have studied and about how people learn and getting students to implement that in classrooms. I mean, we have these issues both on getting faculty to adopt certain practices, but now we're talking about getting students to adopt certain practices. Here, we're going to talk about helping students improve their metacognition and learning.

The university is a lab for you. Your students are going to be your lab, and I do want to talk about equity gaps or opportunity gaps as we move forward. That's a critically important issue. As you said, if we really want to have a diverse workforce, we've got to start when they walk through our door at universities, if not before. But certainly once they come to the university, it's our job to help all of our students succeed.

All right, so let's get to your paper. So we know that retrieval-based learning is impactful and better than reading and rereading material. So if we have a solution that improves long-term learning and therefore will improve student success, then maybe this is redundant, but what's the problem? What is the problem you're trying to address in your paper?

ROBERT ARIEL: Yeah, so the problem is getting this effective strategy into the hands of students. And you mentioned one approach that educators take, is that they can implement maybe retrieval-based activities in their classroom, and certainly that's an effective way to get students to engage in retrieval-based learning.

But as educators, we often talk about teaching students how to learn and making students lifelong learners. And in order for us to really do those things, we need to teach students how to learn. So a lot of learning happens outside of the classroom. So what we really want students to do is to take these strategies, use them in their psychology courses, use them in their biology courses, use them in all their courses, because that's a little bit more transformative.

You can get them to change the way they acquire information, whether it's in a classroom or outside of the classroom. And this is the approach I've taken to training retrieval practice. I mean I'm doing some of the work we're going to talk about now, but I'm also doing work with older adults doing the same kind of things and trying to teach them how to learn information about the medications they're taking to manage chronic illnesses, so it's sort of the same idea.

Here, my goal is, well, can we give them this tool and will they use this tool to actually regulate their learning? Because it's actually not a trivial—it's not a trivial question asking whether they will actually use it. First off, let's just talk about how students use their table and whether they do.

We know that they use it somewhat. So you might have had the experience with students in your classroom using flashcards to study. And you want them to do that. But that's not the strategy they adopt the most. What students tend to prefer to do when they're studying material is mass repetitive reading.

That's a strategy that can be effective in the short-term, but they don't really maintain any information long-term. And what that strategy tends to do is it inflates their confidence in the material. So if you like read your text and read your notes and then you reread it again and reread it again, that second or third time you reread it, it's more familiar, you read it a little bit faster. And people interpret that fluency during reading as a sense of understanding.

And fluency isn't always diagnostic of understanding. In fact, a lot of strategies that are effective signal disfluency, because they're hard. So [? Bob ?] [? Weir calls those ?] Desirable Difficulties. So these are any cognitive strategy that's difficult but tends to improve long-term retention of material.

And it creates this metacognitive illusion often because that difficulty is interpreted as disfluency, and disfluency is assigned to the learner that they're not making progress when they actually are. So they often don't adopt retrieval, so we want to get them to adopt retrieval-based learning strategies.

And also what we want to do is teach them like the most optimal way to do this. Because when they do use retrieval, it turns out that they use it ineffectively. Students adopt what we call a one-and-done strategy. And that's because they're not using retrieval as a learning tool, they're using it as a tool to assess what they know and what they don't know.

STEVEN ROBINOW: Right.

ROBERT ARIEL: And this is how students and educators think about retrieval and think about testing. We think about testing as a way to assess whether we know something. But it's a powerful learning tool. With that knowledge, we should use retrieval a little differently. So what students do with the idea that, hey, I'm using retrieval to monitor my learning, when they think they've successfully recalled it—and I use that word think because they're not always accurate about when they're monitoring the responses they retrieve.

STEVEN ROBINOW: Right.

ROBERT ARIEL: But when they think they've recalled it, they tend to drop it from learning. So if they're studying their flashcards, they might move it out of the pile and continuing studying.

STEVEN ROBINOW: Sure.

ROBERT ARIEL: And that's ineffective because we know, based on research from Katherine Rawson and then John Dunlosky, is that during a single study session, the most optimal strategy you can use is space-repeated retrieval practice. And specifically what you should do is you should recall information to a criterion of three.

And what I mean by that is keep it in the stack of flashcards and keep recalling it until you successfully recalled it three times spaced across time, meaning like if you recall it once, put it at the back of the deck, maybe shuffle the deck up, too, so it's a little random, and come back to it again and recall it again. And once you've remembered it correctly three times, then you can be done learning it, persisting past that one-and-done, which is what students adopt.

So the study that you asked about, that's the whole goal here, is like how can we get people to recall information to a criterion of three? And can we get them to do that with a simple intervention? And our intervention essentially involves strategy instructions. Based on recommended guidelines in the literature, though.

So there's two things. You don't want to just tell somebody to do something, you need to give them a reason for doing it, so you need to explain why it's effective. And you also need to be specific about how to use it. I developed this simple intervention with Jeff Karpicke that involved drawing learners' attention to the learning benefits of retrieval practice, and when I say that, we just tell them that hey, this is more effective than a strategy you typically use, restudying.

And the study you mentioned, we actually showed them some data. So just a simple graph that we told them was from Purdue University students showing the benefits of repeated retrieval practice over restudying. And it turns out there's no deception there because that was actually real data from a previous study with Purdue University students.

So we're just showing them how their student body performs when they use this strategy. So we're saying, hey, this is better, and here's how you should use this strategy. You should continue practicing retrieval until you've successfully recalled the items—each item three times.

STEVEN ROBINOW: Right.

ROBERT ARIEL: That simple instructional intervention improves the way they self-regulate their learning from material. People retain more information than they would otherwise retain using their normal strategies. Interestingly, we also found that they showed evidence that they transfer that strategy in learning new material.

So what we did in our study is we brought students back a week later to the lab and gave them a new set of material to learn without strategy instructions and just allowed them to make decisions about how they want to learn it.

And I don't think I described the methodology, really, but in this task what they're doing is that they essentially have complete control over how they learn a set of material. So they can decide when they want to study it, they can decide when they want to practice testing, and they can decide when they're done.

STEVEN ROBINOW: While they're in the laboratory setting.

ROBERT ARIEL: Yeah, while they're in the laboratory setting. So, I mean, essentially we've created this virtual class flashcard environment that they could do this in. So they had a stack of cards.

STEVEN ROBINOW: So hang on. I just—I'm going to say one thing here. If you're sitting at a table and you have a deck of cards, pause it, go grab a deck of cards because it might be helpful, because it's complicated, it's a little complicated to understand. Once you understand it, it's simple. So go ahead. Thank you. So please explain.

ROBERT ARIEL: Imagine you have the hand—the deck of cards in your hands here. And these decks of cards have your to-be-learned material. So what we had students do is just plan out their study blocks. So choose which cards you want to study, choose which card you want to practice retrieval, and choose which cards, you don't want to learn anymore. And they have complete control.

So they can study all the cards, they can test themselves on all of the cards. And importantly, the task continues until they've decided to drop all the material from learning. So there are multiple blocks of this planning and practice. So you plan a block, you engage and study your retrieval practice based on your choices, then you plan another block, engage in practice again, continue until they decide they're done.

So this is what a student will be doing when they're studying for your exam—probably the night before the exam, unfortunately. But this is what they might be doing when they're making decisions about how to study their content. But we're measuring that on a decision-by-decision basis and looking at the influence of a simple strategy or intervention on those kind of decisions.

STEVEN ROBINOW: So just to clarify what the students are actually doing. So they have a deck of cards in their hand. They're like flashcards.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: And they look at a card and then they have three choices with that card. They can either put it in a pile that they're going to study again, they can put it in a practice test pile, and they can put it in I'm done with this pile. They can put it in three different places. And so maybe you should describe what those three piles are exactly, because what you're monitoring is where do they put each card and when does it end up in the done pile? Is that right?

ROBERT ARIEL: Yeah, exactly.

STEVEN ROBINOW: OK. So take your time to—it's complicated. Go ahead.

ROBERT ARIEL: Yeah. So what each of those piles do is if you choose to study an item, then during the practice block, you are allowed to engage in what we call self-paced study. Essentially the material appears on the screen and you can review it for as long as you want. That's the self-paced aspect of it.

So you have control over how much time you allocate to studying it. So that's typically what we think of as studying. They're rereading, they're adopting whatever strategy they adopt when they're viewing it, which turns out we're not actually measuring their specific encoding strategy, but we're allowing them to review the concept at their own pace.

If they chose the retrieval trial or the test practice trial, they engage in a practice test trial for the item. We've done this with a variety of types of materials, sometimes key term definitions from psychology courses. In the paper you're mentioning, it was Lithuanian-English translations or Swahili-English translation, so it's learning foreign-language vocabulary.

So on a test trial what you would see is you would see the foreign translation, and you'd have to recall the English translation of the word.

STEVEN ROBINOW: OK.

ROBERT ARIEL: So you're practicing retrieving it from memory. Typically in this kind of procedure, what you see is that students' default strategy is to study material early, shift towards testing later, but they terminate study as soon as they move the information through the done pile after they've correctly recalled it once.

So that's that one-and-done strategy that we talked about earlier where when students recalled it once, they think they know it, so now they move it to the done pile and they think they're done learning it. And again, the goal of the intervention is to prevent them from doing that. So keep it in the stack and continue—practice repeated retrieval practice until they've successfully recalled it two additional times.

And it turns out, it doesn't take much more time to do that, because after they've recalled it once, the probability of them recalling it successfully in the next trial is pretty high. You're talking about an additional five minutes of practice here that results in a huge increase in performance.

STEVEN ROBINOW: Over a—how long typically would the study session be?

ROBERT ARIEL: So these study sessions were approximately an hour, an hour long. Well, the practice session was a little bit earlier, actually. So the entire session is an hour, they spend about 40 minutes studying, and the rest is a retention interval.

STEVEN ROBINOW: So 40 minutes plus or minus five minutes or 35-plus—whatever. OK, so that's—five minutes is a—not that much more time given that. So let me summarize and see if I understand it. So a student looks at a card that has an English and Swahili word on it.

And if they're traditional, they'll stick it in the study pile, they'll look at it, they'll stick it in the study pile, they'll look at the next card, put it in the study pile, put in the next card. And then they'll pick up the study pile and do the same thing. Look at it, stick it in the pile.

Once—a typical student, once they might get comfortable with it, they might throw it in the practice pile, and in the practice pile, they get the Swahili word, they have to come up with the English word, and if they get it right, then they'll throw it in the done pile.

And they know what's right because in the test one—and I think this is important, they put in the English translation, and then the computer tells them the real word, the real translation so that they see the correct answer again. And they can determine whether they got it right or wrong, and then they can either put the card back in the study pile or the retrieval practice pile or the done pile. And once it's in the done pile, they're done.

ROBERT ARIEL: Yeah, that's correct.

STEVEN ROBINOW: And your goal is to get them to stick it in the retrieval practice—to work it through testing recall at least three times before they dump it in the done pile. That's what you want them to do. That's what you've instructed them to do.

ROBERT ARIEL: Exactly, exactly. And I left out the part about the feedback you mentioned and that's critical. We actually allowed them to decide whether they wanted feedback, and it turns out they always ask for it, because again, they're using the retrieval to monitor. So they recall, and then they always ask, and I see what the correct answer is.

And when they see that, they know and they can compare that it's accurate or not. It turns out this creates an interesting issue when they're using more complex material than translations where there's this one-to-one mapping between the correct and incorrect answer.

Like if you're having them learn more conceptual information, like a key term definition like, what's the definition of confirmation bias? There, it's much more difficult to monitor the accuracy of the responses even when the correct definition is present with their response here, because what students tend to do is believe they've actually recalled it when the response contains partial information.

So if you take like a textbook definition of confirmation bias, it's this tendency to seek out and use information that confirms the existing beliefs or prior decisions. They might produce a response that partially overlaps with some of those words, like seeking out information that you believe. They don't talk about the using it, though, and the application.

So they're missing some parts of the answer that you want them to have, and we see this all the time when students respond to questions in class where they have a gist of it, but they don't have the complete response. They often think that they know it. They're likely to say, yes, I recall it, and move it to the done pile, but they haven't actually recalled it.

So it's like the one-and-done strategy is even more problematic when students are inaccurate at monitoring the accuracy of the stuff they retrieve.

STEVEN ROBINOW: Right. That's the experimental design that you've set up. Tell us about how it worked.

ROBERT ARIEL: Sure it was more effective than we expected it to be, honestly. We improved performance about 10 percentage points. And importantly, most learners seemed like they were capable of implementing the strategy we wanted them to implement.

And again, it's more cognitively complex than we may describe it so far, because if you think about what goes into repeated retrieval practice, again, they have to be able to remember to use a strategy, they have to monitor the accuracy of their responses, they need to keep track of how many times they've successfully recalled an item across time.

It's conceivable that this type of strategy is just too cognitively demanding for learners to use, but it turns out it's not. They were capable of doing it. But about two-thirds of students were using it in the manner we instructed them to use it. Not everybody was, which is often the case with interventions, you don't reach everybody, unfortunately.

STEVEN ROBINOW: Right.

ROBERT ARIEL: We don't know why they're not using it. Is it a lack of motivation, or is it just did they not have enough time to get to the criteria that we wanted them to get to in our study session? Regardless, though, we're making a dent here, and we're making a dent using a very simple intervention that really would take no class time for instructors to implement.

So, I mean, if you have learning activities for your students, you could just embed these strategy instructions into them. I mean, what I always do is my first lecture, syllabus day, I lecture on, here's how you should study material and here's how you can use retrieval practice effectively.

So I'm essentially giving them the instructions from our intervention during class time and reminding them then when it comes time for my exams, I'm like, hey, remember, this is a highly effective strategy. Because I think we take for granted students' ability just to learn that on their own.

So, I mean, how would you ever learn that retrieval and specifically repeated retrieval is something that enhances your learning? Because you need to be able to attribute that strategy to gains and performance. And there's no way you're going to do that from experience alone. So here's a place where I think they need direct instruction in order for them to acquire the strategy we want them to be using.

STEVEN ROBINOW: Right. And your performance is only measured maybe once or twice in a semester if you're in certain classes. If you're in classes that have a midterm and a final, you've got two points to measure your learning and you have no intermediate points to do that if you don't do it yourself and if the instructor isn't structuring the class in a way that provides practice.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: Students have been successful because they've gotten to your class. So they have some illusion that they do know how to learn. And at some level, that's supported. They're there.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: So they've made it to you that they do have some amount of success, so at some level, their methods work. But you have students that don't succeed in your courses, and of course, we have students who never get to your courses. So it's not just when they walk in there as first day as freshmen. We really need to move this back so that more people have the opportunities to succeed to get there just to get in the door.

ROBERT ARIEL: Right, yeah. And I think that's something that's powerful about retrieval practice is that there's a lot of evidence that it's effective for many different kinds of learners. So we see these are effective for undergraduates, are effective for preschool students, elementary school students, older adults, learners with ADHD. It seems reasonable that there's going to be a lot of bang for your buck for this kind of intervention in the classroom because of the evidence for transferring across people and also types of material.

STEVEN ROBINOW: And as you're saying, that I'm thinking students learning a musical instrument or learning a sport, it's exactly the same, this is exactly what they already do in those paradigms. You have music lessons, you practice, you practice, you practice, you improve, and you can see your improvement over time because you can hear it or your parents come in and say, oh, that sounds so much better.

Or you finally score a goal or whatever it is. You can monitor—it's a different way to self-monitor and you have these moments of assessment. People talk about muscle memory. Well, it's not muscle memory, it's in your head. Your muscles aren't remembering, it's your brain that's remembering.

ROBERT ARIEL: Yeah, that's an excellent analogy, and it's just—again, it's just practicing with your brain. The task we want them to do, just like you mentioned in sports, so the task you need to do in basketball is to make the free throw or to make the three-pointer. And to do that, you practice doing that. So if the task we want students to do is be able to recall and apply information, then we need to have them practice doing that.

And that practice, we now have decades of research showing that it's highly effective and it's more effective than the default strategies that students use.

STEVEN ROBINOW: I think you talk about this in your paper, that perspective of recall that students think they're there to put information in their head, but really, what they need to be able to do is get it out of their head. It's no good if it's in there and it can't get out. So the recall aspect is important because that's, as you said, that's what you want them to do. It's not just, do you know it? Yes, I know it, can you teach it—or can you do it? Can you shoot the free throw or can you explain this?

And so now there's this recognition that we're changing from dumping information in their head, you have to learn this to you have to be able to recall it. I don't care if you learn it, you have to recall it. Well of course, if you can recall it, you've learned it. So which steps should we really focus on?

ROBERT ARIEL: Yeah. There's a difference between, I guess, believing you know it and then actually being able to recall and use the information. So that's, again, this disconnect again between students monitoring their learning and what they think they know and what they actually know.

And it turns out, retrieval does have benefits for monitoring. In one case, the example you brought up there, when they say they know it, what they might be doing is just basing that judgment on some familiarity with the topic.

But when we retrieve, it provides a richer set of evidence for us to base that judgment now, because it's like, OK, I have something in my head that I can evaluate, and those contents, that recollection is often much more diagnostic of later performance and just relying on familiarity or even some false sense of fluency that they might have during studying.

STEVEN ROBINOW: Yeah, exactly. Students mistake familiarity for knowledge and understanding.

ROBERT ARIEL: Yeah, exactly.

STEVEN ROBINOW: And that's what novices do regularly. It's not uncommon. OK. So now so you've explained your first experiment, which was to give them instruction and have them perform, and they performed better than you thought, and those students—you have a control group, of course. And the experimental group did significantly better, 10% or so better than the control group in their performance. So go ahead and talk about the second experiment.

ROBERT ARIEL: Sure. So the question about the second experiment was, will people transfer the knowledge about the strategy they learned when learning a new set of material? So what we did is just bring them back a week later and allow them to study a new set of material. So like in session 1, they studied Lithuanian-English translations; in session 2, they're studying Swahili-English translations.

So now there are new materials. The question is, how do they regulate their learning from the material without instructions using the same paradigm? And it turns out that people who we gave the strategy instructions in session 1 utilized that in session 2 without being prompted.

So, I mean, that's some evidence for transfer across time—across a week. And note, there's no motivation for them to use a strategy in session 2. In fact, I mean, they're incentivized to not use that strategy because they can leave the experiment sooner. So it's surprising that they utilized it even a week later when simply all we did was told them that like hey, this is an effective strategy and here's how you should use it. They did it again.

STEVEN ROBINOW: You told them that in experiment 1—in experiment 2. You didn't give them the instruction.

ROBERT ARIEL: Yeah. In experiment 2, we didn't give them any instruction.

STEVEN ROBINOW: They just come in the room and sit down and do the test?

ROBERT ARIEL: Yeah. So there's evidence there that, hey, they're transferring use of this strategy to a new context, which is what we want. Because we want them to use these strategies in all of their courses and not just when learning—not in isolated contexts. So that's the goal here is to transform how they regulate their learning across all their academic fields.

STEVEN ROBINOW: So now you have this 2018 paper that documents that a simple prompt in the situation you set up in the lab, students—most students, not all—most students will adopt the practice and do well. And so actually, the learning gains, the 10% learning gains in the prompted group includes those that don't adopt the practice. So they're bringing the average down.

ROBERT ARIEL: Yeah, exactly.

STEVEN ROBINOW: If you excluded those that didn't implement retrieval practice in the way that you talk to them, if you took them out, the learning gauge for the subset, the two-thirds, roughly, that adopted the practice would even be better.

ROBERT ARIEL: Yeah, they would. They would be even higher.

STEVEN ROBINOW: So a third of the students don't adopt, pulling the results of that test group down. So really, if you pull them out, oh my God, and have more than a 10% learning gain, there aren't many interventions that I know of where students can do that so simply.

ROBERT ARIEL: Yeah. And again, we were surprised by this, and we've since done research using more complex material. We've even done some study in the classroom using a similar kind of procedure just to investigate how do students regulate when they're actually studying for an exam they have coming up.

So I did this work with Jeff Karpicke again and Ludmila Nunes at Purdue University. We essentially created that same kind of flashcard environment that we talked about earlier and gave it to students before studying for an exam just to investigate how they were studying. It turns out, they did exactly what we say they usually do, which is they use the one-and-done strategy for the most part, though there are individual differences in how much people practice retrieval. And those differences were predictive of their exam performance.

STEVEN ROBINOW: Hmm.

ROBERT ARIEL: So again, just showing that benefit of repeated retrieval practice for actual academic performance in the classroom. And if you look at what's not predictive, what's not predictive is how much time they spent studying, the number of restudy opportunities they choose. But what is predictive is the number of retrieval practice trials they choose and the amount of successful retrieval for the material.

STEVEN ROBINOW: So that's fascinating. I want to focus on that for a second. So can you describe what you did in the classroom? So this was in a classroom. And can you describe a little more what you did and what instructions you did or didn't give students and how you monitored their performance?

ROBERT ARIEL: Yeah. So actually in this study, there were no strategy instructions. Here, we're just interested in, are the claims we're making in the lab about how students regulate their learning? Do they translate into a more naturalistic setting. So like are they actually using this one-and-done strategy?

Because we've documented that in our study in our group that doesn't receive the strategy instructions. And also others have documented other similar things in the lab. And in fact, if you ask students how they study, they basically report that they use something like a one-and-done strategy.

So we were interested in like, do they actually do this when you're looking at their actual decision-making behavior when studying for an upcoming exam? With the idea that maybe down the line we can sort of implement our intervention in the classroom, and that's kind of the next step which we haven't done yet.

But what we did is working in Ludmila Nunes's cognitive psychology course, she gave each of her students an assignment before the exam where we said, we just would like you to study this set of material from the topics we've covered over the past month and you can study it however you want. No instructions about how to study it. So it's just like you should use these materials to prepare for your exam. Because what we were interested in is, again, what decisions do they make when studying it?

So what they got was the same flashcard environment that we used in the study we described earlier, but now tailored to exam content in that cognitive psychology course. So they were all key term definitions. Again, like what is confirmation bias? What is an algorithm? So that would have been from her judgment and decision-making session of her course.

And here, it's a little bit different because what they need to recall is the definition associated with each key term. But we're using factual and applied questions that she would ask on her exam, but not the same questions.

STEVEN ROBINOW: Right. But they're doing this on a computer so it can be monitored.

ROBERT ARIEL: Yeah, they're doing it on a computer so we can monitor their decisions. And it's the same idea. So they have a stack of flashcards containing these key terms. And they take the stack and they move some to a study pile, they move some to a practice testing pile, and then they move some to a done pile when they're done. So again, we're just tracking your decision-making and studying how they learn the material in preparation for the exam.

And of course they're doing other things when they're preparing for the exam, but we told them to use this environment to do it. Because we're interested in like, does how people self-regulate their retrieval practice actually influence grades? And it turns out it does. People who chose to test more in the flashcard environment perform better. So those test choices correlate with exam grades. Study choices don't.

Again, time spent studying doesn't. That's important because I think we often give students bad advice when they perform poorly and we tell them, well, you need to spend more time learning. And we know time is not a mechanism that produces learning, it's all about what they do with their time.

So you're telling them the need to spend more time is kind of trivial. It's not telling them really what they need to do to learn and retain information. I mean, you should do like we mentioned and tell them they need to adopt specific strategies like retrieval practice.

In this task, they made those decisions, and again, it was predictive of performance. So the next step is taking what we do in the lab and seeing if we can get students in the classroom to use these strategies. I'm confident that they will be effective.

STEVEN ROBINOW: So how are you going to do that? When and how are you going to do that? And do you have the IRB for it?

ROBERT ARIEL: I don't have the IRB for it yet. So I've been doing a little bit more basic research, and of course, the pandemic has made classroom research a little bit more difficult.

STEVEN ROBINOW: Sure.

ROBERT ARIEL: Soon. But actually, I think the intervention we'll utilize is going to be simpler. It's just going to involve—I think you might need to hit them over the head a little bit more, but keep reminding them to use the strategies. So like in the lab study, we were interested in if they would spontaneously transfer the strategy.

Here, I think we want to just keep reminding them, and again, nudging them towards using it. I think for the next study, it's just going to be implementation of that same intervention but in a classroom using that flashcard environment. So it's like when it comes time to study for the exam, give them access to some online learning environment where they can make decisions about how to study and investigate the effects of those strategy instructions versus no instructions.

STEVEN ROBINOW: So that's interesting. So in a class, you've got one class, how are you going to divide the two groups? And do you actually need to? So if you have a situation—so you've already done this test in a class a little bit.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: And you know that a third of the students, you tell them and they don't do it anyway. So do you need to have a control group where you don't instruct them or do you let students—do you instruct everybody knowing that you're going to get 30% that don't instruct and use them as a control for they really got the same thing and they made a choice, an active choice not to participate, not to do what you said? So can you set up a criteria to set up two groups?

ROBERT ARIEL: Yeah, that's an interesting question. And, I mean, obviously we want to identify why people aren't using the strategy. So the subset of people that don't seem to be adopting and we don't know why they're not using it. And it could be for a variety of reasons. Yeah, but in terms of should you randomly assign, I mean, I think this kind of design allows us to randomly assign. Still, I mean, the question, is that an ethical thing to do?

STEVEN ROBINOW: That's where I'm headed, because you've got a practice that is impactful. And where is the ethical line? When do you stop collecting data using a control group and just say oh yeah, that's not ethical anymore. We need to—everybody needs this instruction.

ROBERT ARIEL: Yeah. I mean, the scientist in me wants that randomized controlled experiment and we could practically do it delivering the same intervention we've done in the lab in the sense that the intervention was just something they read. It wasn't even provided by the experimenter because in our task we try to standardize instructions. So they read all the instructions on a computer screen, which is something they read on a piece of paper.

So we certainly could easily give half the students those instructions and half not, and the scientist in me wants to do that. Though my classroom is probably already contaminated because I'm always talking about the benefits of these. So that might not be the best approach anyways.

I mean, the best approach might be to—I could potentially even just give them the platform that I've used at Purdue. It certainly would be interesting to just see if my students behave differently than the standard students because I'm always talking about the virtues of retrieval and giving them these instructions. So maybe in theory they would show different patterns than students at other universities, or at least you would hope.

STEVEN ROBINOW: Right. But if you gave them to students or one of your colleagues at Wesleyan and in a biology class or pick a discipline where they haven't had a psychology classroom to talk about, to hear about this to, you're right, because you've got this contamination problem.

ROBERT ARIEL: Yeah. I think what I'm waiting for this project is an NSF grant.

STEVEN ROBINOW: There you go.

ROBERT ARIEL: You mentioned biology here. I think that this is the type of project that would be wonderful for an IUSE Grant, Improving Undergraduate STEM Education, and collaborating—

STEVEN ROBINOW: Absolutely.

ROBERT ARIEL: —with STEM scholars and investigating ways to implement these strategies effectively in the classroom. That's the next step here. I mean, I'm an experimental psychologist and always start in the lab and then gradually scale up and that's my approach. So you've kind of got us in this transition phase where the next step is scaling into the classroom.

STEVEN ROBINOW: So let me throw it out there, and this may or may not stay in the recording, if you need to be connected with biologists who are interested in this sort of work—

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: I mean, there are people out there who would jump on an opportunity. Super interesting. Anyway, all right, so let's get back to—we know what you want to do next. You want to bring this into a classroom situation and really, really nail this down. And I would think one good-sized experiment, if you want the randomized control—to do the randomized control, you want to do it once and that's it, because then it becomes ethically not tenable.

You have to then just say you have to do this for everybody. And then the question becomes, how do you bring that 30% that don't adopt? And that's a super interesting question. Why don't they adopt? Is it time? Is it that extra five minutes?

ROBERT ARIEL: So recently what I've been thinking about is just how do you make this strategy more valuable to students here? And are there ways to co-opt retrieval and to enhance the benefits of retrieval by making the retrieval aspect more meaningful to the students?

And here, building off motivational research and this idea that they really need to value a task and maybe like some of the barriers of implementation is just that they just don't value it. So like how do we make retrieval more meaningful? Something I've been doing in my classroom recently is using these writing prompts that are a little bit more person-centered.

So like I mentioned before like we might ask somebody to practice retrieval on the definition of confirmation bias. Well, you can ask them to practice retrieval in a slightly different way. Like you can ask them to recall an experience where they displayed confirmation bias, which still requires them to recall what confirmation bias is, but then also tie it to themselves.

That's perhaps beneficial for a variety of reasons because we know that self-reference also enhances memory. So any time you get people to process information in relation to themselves, there's memory-enhancing benefits of that. There's some research suggesting that increases interest in motivation because you're making the materials seem more valuable.

So co-opting these retrievals in ways that tie more to students' everyday experiences and their personal experiences might be one way to get them to enjoy doing it more and to value it more. I had an assignment in my last cognitive course that was built around this idea, and students seemed to really like it. So now the question is like, OK, let's test it in the lab and see if this provides benefits above and beyond typical retrieval.

STEVEN ROBINOW: Oh, I love that. So the notion of them tying it to a personal experience which does lots of things. So when they think about confirmation bias, they can recall their personal experience that they've done and then they can pull up the elements that you want, which are what the elements of the definition because it's not simple, it's complicated. That's fantastic, I love that.

But let me tear away from that. When you talk to people about retrieval practice, you can show them data, you can show them data, you can show them data, but data are less important than belief.

How do you get students to go from the data that you want to show them or the experience that you want to give them, and how do you get them—and I like your idea of really personalizing this stuff. So how do you get them to believe it so deeply that it just becomes part of what they do? Because it seems to me like—this is fascinating to me, that really, belief becomes where it's fully integrated.

ROBERT ARIEL: Yeah. And it's hard to change people's beliefs.

STEVEN ROBINOW: Oh my goodness, yes.

ROBERT ARIEL: Yeah. And so what motivated the intervention that I developed was a series of studies by other researchers like [? Tulse ?] and Benjamin where they looked at whether people can learn from experience that testing is beneficial to memory, and people had difficulty doing that because they never make that attribution that testing improves learning.

In fact, I've done a study myself with Jared Hines and Chris Herzog where we just ask people to forecast their memory for material after four study trials versus after four test trials. And if people think they're going to repeat the study, they think they're going to increase learning across time, they think they're going to be tested repeatedly, they predict flat learning.

So that tells you what their belief is, like they don't believe this is effective strategy. So how do you do that? Just giving them the data work? I mean, in our case, it worked, but we don't have any evidence that we actually changed their beliefs.

It turns out, that second experiment that we talked about earlier where we saw transfer in the second session, we don't know if that transfer is due to the strategy instructions or even like the experience of we told them it's successful, they experienced it also being successful, so that combined with the knowledge might have consolidated the idea that yes, this is an effective strategy.

You told me it would work and it actually did, because we tell them some things are going to work and they don't. So here, they were successful after using it, so maybe that contributes to why they changed their beliefs. So that's potentially important, but again, making it more valuable might be another way. So getting them to—I guess two things. You want to make it more valuable and tied to their personal experience, and you want to reduce the opportunity costs of implementing the strategy. Because it's effortful.

STEVEN ROBINOW: Right.

ROBERT ARIEL: And that's one reason why students might not adopt it, because it's a little too hard.

STEVEN ROBINOW: It's too hard.

ROBERT ARIEL: Yeah. Sorry, I kind of got off-track there at the end because my cat broke into the office.

STEVEN ROBINOW: Oh. OK. In experiment 2, they come back into your lab and they take a similar-style test with cards unprompted.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: So they're triggered as well. I mean, this is where they learned—they heard that instruction, they come back in, you don't know what's thinking. It would be interesting to put them in a different—somebody else's class to collude with somebody unknown to them, that they go into their class they do a different format of study and then ask, are they using retrieval practice now where they weren't before?

If you had some other situation where you could monitor them somewhere else that's completely unaffiliated so they're not triggered by coming back into your laboratory and remembering what they did last time. Because, as you said, you don't know that they're going to take that out of the classroom and use it to learn their amino acids in their intro biology course.

ROBERT ARIEL: Yeah.

STEVEN ROBINOW: Which might be very analogous work.

ROBERT ARIEL: Yeah, that's a great idea.

STEVEN ROBINOW: Cognitive psychology and experimental stuff is fascinating to me now. This is where I spend all my time. All right. So next you're bringing us—you're moving this out to the classroom next, really, that's your goals, and hopefully you're grant-funded in the near future to do that, right?

ROBERT ARIEL: Yes.

STEVEN ROBINOW: It's fascinating work, and it's—I think this notion of moving away from what faculty do and getting what students do. I mean, faculty spend—in a semester time, you've got 40 hours with a student in a semester in a class and it might be 300 people in the class. And so your personal time with them or your time to schedule their activities, well, you've got 40 hours however you want to use it.

So you could use it for retrieval practice and maybe that would be how we should use all of our time, because they can study the material outside as well.

ROBERT ARIEL: Faculty could adopt some of the ideas we're talking about. So this idea of using this criteria learning and this magic number three. A lot of faculty are adopting retrieval practice in their classroom using things like low-stakes quizzes. And that's great, but you can also incorporate some of these ideas in those quizzes.

So you can design your quizzes to promote space practice so that they're recalling from previous lectures. And just like students don't drop materials—so sometimes we quiz students on content once. So it's good to quiz them on content repeatedly. I mean, you can use different forms of the same question.

But the idea is that different varieties of question targeting the same kind of knowledge with the goal you want them to engage in repeated spaced-retrieval practice, because that's, again, better than just testing yourself on it once. So I think educators are adopting these strategies and they're doing quizzing in the classroom, but they can take some of these ideas about how we know about what's the optimal way to schedule retrieval and maybe tweak what they're doing just slightly to get a little bit more bang for their buck.

And we have all this technology now that makes this easy, such as clicker systems. I mean, it's an easy way to deliver a retrieval practice into the classroom.

I mean, I like to use it in my classes, and what I love about it is that you get immediate feedback. So if you give students a multiple-choice question, you see what percentage got it correct and then you see what types of mistakes they make so you can immediately address misconceptions and use that as an opportunity to draw attention to why they're making certain errors and what those concepts really mean.

So it's a powerful learning tool in the classroom that we can utilize. Certainly some educators are utilizing it already. But again, my recommendation is like, hey, take what we know about optimal schedules of practice and use more repeated retrieval practice in your daily activities.

STEVEN ROBINOW: All right, so I like that a lot. So faculty can do a lot in the 40 hours in semesters that they have with them or 30 hours in quarters that they have with them they can do things in their courses, in their classrooms, or their laboratories to expose them to this and to give them opportunities for retrieval practice.

But for the student, there's a lot to be gained to make the student independent in this process. To provide them the knowledge about, hey, this is really going to be a more effective way.

You need to learn retrieval-based learning, you need to learn about it, and you need to start to understand how to structure it so that you can monitor your learning and ensure that you've learned things effectively for your future. If we really want lifelong learners, they need to do it on their own, and they need to be able to do this work on their own.

ROBERT ARIEL: Exactly.

STEVEN ROBINOW: And not every instructor is going to make a nice class for them that uses retrieval-based learning. And if we did, they'd get through college and they'd go out to the work world and they wouldn't know how to learn things because their life had been structured for them. And they would be oblivious to it, they wouldn't really understand that structure.

ROBERT ARIEL: Yeah, exactly. And a lot of students struggle when they transition to college because they don't have these self-regulation skills. So now they're in an environment where they really have to monitor their time and they have to self-regulate. And we don't teach them how to do this.

So giving them basic skills is going to enhance your experience in your classroom and hopefully beyond, because again, the goal is to get them to adopt these strategies in any of their courses because the evidence suggests that it's likely to be effective across a broad range of disciplines.

STEVEN ROBINOW: Right. Fascinating. OK, so last—I guess the last question—I know we're over. So you're an experimental cognitive psychologist that studies learning and memory. This must impact your view of the world. So I want to ask you, I'll be very specific, I want to know how this affects you when you're at a restaurant or out in the world.

Don't tell me anything about academia. I don't want to hear about how it affects you on campus or your students or your colleagues. I want to know how it affects you when you're out running or doing any activity where you're interacting with people in a completely different environment. How does this perspective change your life?

ROBERT ARIEL: It's a tough question.

STEVEN ROBINOW: Yeah, it is, it is.

ROBERT ARIEL: I think psychology in general, one thing it does, and not just understanding of learning and memory, but just what we know about who we are and psychological processes that influence behavior, is that it just gives you a little bit more empathy for people.

Like, you should empathize better with yourself and empathize better with other people because you understand memory and these other processes and the errors people make. I mean, you mentioned at a restaurant there. That's a situation that's ripe for memory failure to happen. Maybe your waiter or waitress forgets your order. I mean, I guess I don't have a good answer for that question, sorry.

STEVEN ROBINOW: I think discussing empathy is a really excellent answer. If it just makes you more empathetic to the world, I think that's huge.

ROBERT ARIEL: Yeah, and certainly it does.

STEVEN ROBINOW: The world could use more empathy right now. The world could use a lot more empathy right now.

ROBERT ARIEL: We could. I mean, I think that's the strong suit of psychology, and certainly other disciplines have this strength as well, but just making—helping students become more empathetic, helping yourself become more empathetic because you have a better understanding of why people behave the way they behave.

And certainly that makes you more sensitive to issues of social justice and inequity and all these things. I mean, that's, I think, a powerful thing about psychology and a transformative aspect of psychology that can change your experience wherever you are.

STEVEN ROBINOW: Yeah. I don't think we talked about equity and opportunity gaps, but we mentioned them, and I assume that those will become part of your future research to look at how retrieval-based learning in classes impacts equity gaps. I assume that's stuff you'll look at.

ROBERT ARIEL: Yeah. That's something I'm very interested in, especially in the environment I'm working in now because it affords opportunity to study this.

STEVEN ROBINOW: That's fascinating. OK. Robert, I think we've taken up a lot of your time. I want to thank you so much for taking the time to talk with me today. This has been just a wonderful discussion. I've really enjoyed it. I love this topic. I hope you'll come back again to talk some more about your work.

ROBERT ARIEL: Yeah, I'd love to.

STEVEN ROBINOW: Thanks so much. This has really been a fun and super interesting discussion. I've really enjoyed it. Thanks—thanks so much for your time.

ROBERT ARIEL: Thank you for having me.

STEVEN ROBINOW: Absolutely. For more information about Robert Ariel, his research, and favorite books and papers, please go to our website, teachingforstudentsuccess.org. Thank you for spending time with us today. I hope you found this discussion interesting and helpful. Please share our podcast and website with your friends.

Those of us at Teaching for Student Success would love your feedback. Please contact us through our website at teachingforstudentsuccess.org. Teaching for Student Success is a production of Teaching for Student Success Media. Let's end this podcast with some music by JuliusH. Some of Julius's music can be found on Pixabay.

[MUSIC PLAYING]