Episode 22

Connecting Classroom Inequities to Student Performance: EQUIP, a Tool for All? with Daniel Reinholz

>> Steven Robinow: This is Teaching for Students Success. I'm Steven Robinow. We have been singing the praises of active learning for years. For decades, hundreds if not thousands of research articles document the benefits of active learning. However, developing an active student-centered classroom isn't easy. As researchers study the success of students in active classrooms, they expose new questions to ask, they generate new data to analyze. These data put classrooms implementing active learning practices under the microscope. In looking closely at the details of implementation, researchers are now uncovering evidence of practices resulting in inequities, in some and perhaps most active learning environments. Today we talk with Dr. Daniel Reinholz, Associate Professor in the Department of Mathematics and Statistics at San Diego State University. Daniel's doctorate from UC Berkeley is in mathematics and science education. That's one degree. Their research focus is on equity analytics, systemic change, and disability justice. Our conversation today will focus on their paper, When Active Learning is Inequitable, Women's Participation Predicts Gender Inequities in Mathematical Performance, published in 2022. in the Journal for Research in Mathematics Education, with co-authors, Estrella Johnson, Christine Andrews-Larson, Amelia Stone-Johnstone, Jessica Smith, Brooke Mullins, Nicholas Fortune, Karen Keene, and Niral Shah. Welcome, Daniel. Thank you for joining me on Teaching for Students Success.

>> Daniel Reinholz: Hi, Steve. Thanks so much for having me. It's really a pleasure to be here.

>> Steven Robinow: Yeah, I'm excited to talk to you today, so let's start. Let's get right into it. Let's start with the purpose and motivation of this research.

>> Daniel Reinholz: Absolutely. Yeah, so I want to take you back in time about eight years ago, and so this was shortly after I had graduated with my doctorate as a postdoc. Niral, who's the last author on this paper, was a classmate of mine at Berkeley, and we've been colleagues ever since. The motivation for our research has been we want to work on projects that we know are going to impact students in schools and in universities today. We don't want to wait, but we really want to have a practical impact now, and one of the issues that we know has been a challenge in the US, but really around the world, and we've known for a while is inequity, right? And there's a lot of reasons for that in society, but we know that schools can either be a place that can perpetuate that, or what we hope for is to actually disrupt those inequities in our society, and so that set us out to build the EQUIP tool, and so what EQUIP is, is an observation tool, and there's a lot of observation tools in our field, but the thing that we wanted to do different was we wanted to actually be able to look at and answer the who question in a classroom, not just what's happening in the classroom, not is there active engagement or things like that, but who gets the active engagement? What types of questions, what types of depth for different students, both, so we could do better research. We know that there's a lot of qualitative studies that tell us things that happen in classrooms, rich ethnographic studies, but it's hard to do that in 10,000 classrooms. So we wanted to build a tool that could have some of those tradeoffs but still help us understand what's going at a disaggregated student level. And then we didn't want to just document what's happening, but we wanted to find ways to give that data back to instructors, and also help them change their teaching, and that has been a focus of our work for the last number of years.

>> Steven Robinow: Okay, great. So I think you should describe the EQUIP tool. This is an essential element of the research that we're going to talk about, so this is an observation platform that you have developed, so let's spend a few minutes talking about that.

>> Daniel Reinholz: Absolutely. So when we think about EQUIP, there's sort of two parts. We can think about the observation protocol, and then we can think about -- we've developed a free web app that can actually help with the coding as well. So the observation protocol itself, many tools that you've encountered might have sort of a rubric approach. We want to rate things on a one-to-four scale. How active is something? We've taken a different approach with EQUIP, where we just try to quantify events that happened in the classroom, and what we focused on in particular is participation. The reason we focus on participation is that we know from tens of thousands of studies in educational psychology, that there are certain things that can help students learn, and in particularly, we know that when students try to explain their ideas, that's actually more effective for learning than just re-studying something they might have read. And similarly, we know that having students practice doing -- it could be a procedure, it could be a conceptual problem, but having them actually practice the thing they're trying to learn in a low-stakes, low-pressure way, is one of the most effective ways for them to learn how to do it, to supplement what learning they might have done in a classroom. And so given these studies, we know that participation is really important for learning. It's not the only thing that matters, but we know that it's one thing that matters a lot, and it's one of the reasons that active learning classrooms can be so effective for students, because they get to do all this meaning-making and explaining and sharing with their peers. We also know that participation matters, because it helps us see ourselves and identify as a mathematician, or as a scientist, or an engineer, or maybe even as a musician. It could be any kind of discipline that we're working in. At an individual level, when we have that success, we see ourselves as successful, and also at a social level, when we're successful in front of peers, that helps build our identity. That helps us feel like we belong. And so for both of those reasons, sort of the cognitive and the social, we know that participation is one type of thing that's really important for students in a classroom. At the same time, there's a number of studies and a lot of these studies preceded the work with EQUIP, studies by Sacker and Sacker looking at gender in schools, and just noticing that patterns of participation, there was a lot of gender inequity, and that was persistent across different settings over a number of classrooms they observed. And similarly, there's other studies looking at racialized participation in classrooms, and the ways that racial inequities can arise. So we know that there's these inequities that arise in the classrooms, and so what we wanted to do was build a tool that could capture those in a sort of efficient and low-cost way, either for documented research or giving this data actually back to instructors. Just to be really specific how we do that, we look at any time a student participates at the classroom, and then what we're going to do is code around the nature of that participation. It could be, what types of thing the students said. It could be, how much they talked. It could be what types of questions a teacher asked them that led to their response. And so we can sort of capture a number of different things at a student level or teacher level, related to that participation, and that's something we customize across studies or another user can customize to fit their context. For example, in recent work, we've even been looking at all the types of nonverbal participation that someone has. So it might not just be talk, but we know that talk is something we focus on a lot in the research, especially in college classrooms.

>> Steven Robinow: So the EQUIP tool, which stands for equity quantified in participation, in your study, I believe you set up cameras in the back of the room, a camera in the back of the room and recorded the room during the session, but this could also be done with an observer in the back of the room.

>> Daniel Reinholz: Absolutely.

>> Steven Robinow: Okay, great. And so now let's get into your paper. Let's talk specifically about what you were concerned about in math classrooms. Now you've spoken generally about inequities. What was the specific inequity you were looking at or concerned about in math classrooms, and what led you to that?

>> Daniel Reinholz: Absolutely. Yeah. Very early on, my colleague, Niral, who we've been building this tool together, the focus of his research has always been on how race plays out in classrooms, and a lot of my research has focused on how to build something, and so we said let's build something that captures that. As you might know, it turns out, it's actually hard to build an observation tool that works, and so the very first studies were just proof of concept. Can we actually build something and generate valid and reliable data that tells us something about how the classroom experience is different for different students? One of the first studies that we did with EQUIP, I was working with a graduate student at San Diego State, and she had this rich dataset with five cameras in the classroom. Some of the cameras were on four different small groups, and then one was on a whole-class discussion, so this was just a beautiful data set. And we just said let's analyze this and see what we find. And actually, the results were a little bit surprising to us. What we found is that when we looked in the small group settings, there were groups where women were contributing in all these different ways and putting forward these meaningful proofs, and sometimes they were just leading their groups in understanding, but when we looked at the whole class, those same women weren't talking. And we even observed a couple of conversations where the class was stumped, and here we're listening to this side conversation where these two women in the class, they had solved it. They had the answer, but they weren't sharing with everybody else, and sort of the rest of the class was stumped, and so that gave us something to reflect on. We're like, hm, why is this happening? And we dig deeper, and we could kind of start to understand the ways that there was sort of this masculinized environment that actually wasn't safe. It wasn't welcoming for those women to participate in that whole-class setting. And when we dig deeper, into -- even into the small groups in that particular classroom, there were sort of some of these sexist interactions that we were noticing from some of the men in that class, and so all these things added together in a way that just created barriers for some of the students in that class, even though they understood the material, you know, they were kind of working on this in so many really sophisticated ways, but it was the social environment that wasn't allowing that to come out, and frankly, wasn't allowing other students to learn from what they had discovered.

>> Steven Robinow: So that the result of that is twofold. One, the class is held back and doesn't get to benefit from their understanding and their teaching.

>> Daniel Reinholz: Exactly.

>> Steven Robinow: And then secondly, they don't benefit from the pride, the ownership that comes with, "Hey, I have an answer for you, and let me teach this to you. Let me explain how this works," --

>> Daniel Reinholz: Exactly.

>> Steven Robinow: -- which then allows them to see themselves as, in this case, mathematicians, as practicing mathematicians, see themselves in that role.

>> Daniel Reinholz: Absolutely. That's going back. That study was in -- published in 2019 in Educational Studies in Mathematics. Fast forward a few years from then, still not to today, because it's the project that we're going to talk about today, started a few years ago, but some of my colleagues, Christy Andrews-Larson and Estrella Johnson were some of the people working on the Times project and many of the other authors as well, and so the Times project was really quite a profound effort to develop inquiry-based curricular materials and instructional methods for upper division mathematics in linear algebra, abstract algebra and differential equations. This was a huge effort to really take seriously like what's happening in upper division classes in our field. I was having a conversation with either Estrella or Christy. They had sort of this perplexing result. They had built these fantastic curricula, they have a pretty robust professional development process, and they had recruited a bunch of instructors from our field, some mathematician, some math educators, to implement this, and what they found is when they were implementing it, the results of their study was actually that in the inquiry classrooms, there were larger inequities between men and women than in the traditional classroom, and this was just baffling. Like, why did this happen? This kind of flies in the face of the idea of the active learning classrooms that we're trying to build, or I think even sometimes there's a narrative in our field that active learning will fix everything. Students are going to learn more. It's going to be more equitable. If we could just get everybody to do this, all the problems will go away. This sort of conventional wisdom was in opposition to their findings, and so when we were having this conversation, I just had a thought in the back of my mind, "Well, we have a tool designed to look at what's happening in classrooms, and we could disaggregate what's happening for women and men in these classrooms." And so this just felt like the perfect opportunity to take all this amazing work that they had already done and take a new lens to it.

>> Steven Robinow: Okay, so just to review what you said, because I don't want people to miss this, the work that led you to this was one in which they developed these great curricula. They're being implemented, and they're being tested against control classrooms that it's a non-inquiry based classroom.

>> Daniel Reinholz: Yeah.

>> Steven Robinow: It's a traditional mathematics classroom.

>> Daniel Reinholz: Exactly.

>> Steven Robinow: The traditional mathematics classroom, there's no gender distinction. Women are doing as well as men overall.

>> Daniel Reinholz: Mm-hm.

>> Steven Robinow: And then when they implement the inquiry-based curricula, men improve, women don't.

>> Daniel Reinholz: Mm-hm.

>> Steven Robinow: So you generated a well-designed active learning classroom, you trained faculty in how to use it, and you end up with a situation where you have generated -- you've improved part of class in terms of performance, but another part of the class you haven't improved, and interestingly, you've got this split now that -

>> Daniel Reinholz: Yeah.

>> Steven Robinow: -- you've got women not doing as well as men. Okay. All right, that brings you to asking that question and saying, Let us EQUIP to see what's going on in the classroom and see if we can tease apart the basis of this inequity.

>> Daniel Reinholz: Yup, exactly. And now, this was a complicated study, because we were looking at secondary data. If we had designed it from the offset, it would have been smooth, but we just had to do a lot of work to get the data in shape and to sort of do things so that we could analyze it with the tool. If you want to read about those details, they're in the article. I'm not going to go too far into that here. When we analyzed the classrooms, it was interesting because sort of, overall, participation was actually pretty balanced between men and women, but when we split that up by different classrooms, then we started to find there were some pretty large differences, where some classrooms were almost entirely dominated by the men in the classroom, and on the other side, there were classrooms that had a whole lot of participation from women. And so we did a bunch of statistical analysis, and what we found was we could actually create a regression model to find a correlation between how much women were participating, and what types of gender performance gaps we were finding.

>> Steven Robinow: Okay, let's add a few details here. You looked at 20 different classrooms -- classes, all in a higher-level mathematics, at how many institutions?

>> Daniel Reinholz: I think most of these instructors we're all at different institutions, so this is probably 15 or 20 different institutions.

>> Steven Robinow: Okay, so each class represented one instructor?

>> Daniel Reinholz: Each class was one instructor. We had multiple observations of that class, and then there was another instructor at that institution sort of as a comparison.

>> Steven Robinow: Right. So 20 different instructors, plus control classes that were non-inquiry-based.

>> Daniel Reinholz: Yup.

>> Steven Robinow: Great. And at a range of institutions.

>> Daniel Reinholz: Mm-hm.

>> Steven Robinow: These are not all are R1s, these are not all four-year comprehensives there.

>> Daniel Reinholz: No.

>> Steven Robinow: Yeah. Or and not all community colleges.

>> Daniel Reinholz: And so --

>> Steven Robinow: Any community colleges, since these are upper division math?

>> Daniel Reinholz: I don't think there were any community colleges.

>> Steven Robinow: Okay. Probably no community college with upper division math. That wouldn't -- yeah, okay. All right. I just wanted to fill in that little --

>> Daniel Reinholz: Of course.

>> Steven Robinow: -- blank. Okay. So we got 20 classes, and you see differences, so tell us about the breakdown there of those data a little bit.

>> Daniel Reinholz: Yeah, so I think the first piece that we found was sort of just the performance, and then 15 out of those 20 classrooms, you had men outperforming the women in that classroom, and so that was sort of the base of what we were looking at, but just to remind folks, on the whole, students were still learning more. It's just that some students were learning even more than men, which was creating the inequity. So I don't want to give the false impression that oh, we should just lecture at everybody, because the classrooms were still effective overall, but we need to figure out ways to equalize those inequities, or make it more equitable.

>> Steven Robinow: Hang on a second, because there's a comment you make at the beginning of performance outcomes. Men in the inquiry or in classrooms significantly outperform men in the non-inquiry classes.

>> Daniel Reinholz: Mm-hm, correct.

>> Steven Robinow: We found no significant differences in performance for women between inquiry and non-inquiry. So the classes in general did better --

>> Daniel Reinholz: Right.

>> Steven Robinow: -- but those improvements in performance are due to the men doing better --

>> Daniel Reinholz: Right.

>> Steven Robinow: -- not the women's performance.

>> Daniel Reinholz: Right.

>> Steven Robinow: Okay. Okay.

>> Daniel Reinholz: But another way to think about that is that the active classrooms, they're not harming anybody, but we need to figure out ways that we can shift that balance of participation to make it a more equitable environment so that all students in those classrooms are actually benefiting from some of these new learning strategies.

>> Steven Robinow: Right. Women aren't doing worse in the inquiry-based classrooms than non-inquiry. They're doing the same.

>> Daniel Reinholz: Exactly.

>> Steven Robinow: They're not being harmed, but they aren't getting the benefit that men are experiencing.

>> Daniel Reinholz: Right. Right. And I should just be clear. This was just one study that we did, but we looked at, you know, there's a number of other field studies that we talk about, some larger studies, randomized controlled trials, and I think all of them were -- most of them were in K-12. One of them was in like a military academy, but they found basically the similar result that the active learning classrooms were improved performance overall, and I think in some of the classes, there was also performance gains for women, but they just weren't as large as the ones for men.

>> Steven Robinow: Okay, so now let's talk about why that is. What did EQUIP -- what did your observation tool help you decipher?

>> Daniel Reinholz: Yeah, so what we noticed was basically, when we were modeling this, in the classrooms that where there was essentially no participation from women or very little, there was about a half a standard deviation of difference in the outcome ventures, and as we looked at classrooms where the amount of participation from women was higher, and it didn't have to be a lot higher, it would actually get closer to either the same or there were some classrooms where women were outperforming the men. And so what we found was really, these were smallish classes overall, and so there was about an average or like six or seven women in most of them. If we were to think about increasing that performance in a way that would actually reduce that half of a standard deviation gap, it would really -- that only amounted to about 60 or 70 times that women were participating over the course of a 145-minute unit. And so when we think about what we we're looking at in the classrooms, that wasn't something that was actually that hard to achieve in practice. It wasn't that much participation that was sort of differentiating the classrooms at one end of the spectrum from the other.

>> Steven Robinow: Now, 60 or 70 moments sounds like a lot, but in the way that these inquiry-based classes are running, how many interactions are there? How many counts are there in a typical 145-minute session?

>> Daniel Reinholz: Yeah, many of the classrooms in this study had 100, 150, 200. Some had more, and so it really amounted to like, there was a lot of participation, but to be clear, we don't have a causal model. We can't say if you just make women participate more in this context that it's going to reduce the inequities, but we do know, I think, from a theoretical reason, but also, based on this data, that the participation is one key factor that's relating to the inequity.

>> Steven Robinow: And what EQUIP is counting is every time a student talks. It doesn't matter for how long. It doesn't matter -- I mean, you do have different ways of coding how -- what the interaction is like, but just the participation alone, if a question is asked and a student responds, a student responds, --

>> Daniel Reinholz: Exactly.

>> Steven Robinow: -- says something, and if another student responds to that, that's another participation.

>> Daniel Reinholz: Mm-hm. Exactly.

>> Steven Robinow: Every time students participate, EQUIP counts it.

>> Daniel Reinholz: Yeah. This is sort of a simple model here, right now. What we were just looking at is, is participation, and I guess one other detail to look at also is the way the nature of this secondary data set, we're sort of looking at overall participation from women, and overall performance for women in the class, but we actually don't have a way to tie the data for any individual student. Right? So if we did -- if we were designing the study from scratch, we could have a much more robust design. We would use some sort of hierarchical linear modeling. We could control for things at a student level, and that's actually something that we've done in some other studies, but that we couldn't do in here due to the nature of the data.

>> Steven Robinow: Right. Now, with these 20 classes, like you said, 15 of them generated a gender gap and gender inequity. Five of them did not. In five of them, women performed as well as men or better, as you said.

>> Daniel Reinholz: Yup.

>> Steven Robinow: So you do -- in part of your analysis, you break these data down into sort of three groups. You talk about three groups. You have a centralized inquiry class, you have a shared inquiry class, and a distributed inquiry. You sort of you sort of take these 20 and you say, well, it looks like they break down into sort of these three categories, and you can look at performance. You want to talk about those a little bit?

>> Daniel Reinholz: Yeah, let me talk about that a little bit. And so the reason that we did this, these breakdowns, is I think we have a lot of studies where we run a bunch of fancy statistical models, and that as a reader, you really don't know what's happening. You don't know what the classrooms look like, and so we wanted to give folks some sense of what these classrooms looked like. Now, this is an approximation. We can't say these are super strict categories. They're a little bit overlapping, but what we tend to notice was that in the classrooms that had the most inequity, they were still inquiry-based. Instructors were diving into student thinking, but there was much more of this back and forth directly between the instructor and maybe an individual student for a while, or a couple of students, and I think actually, many of us might be a little bit familiar with that scenario, where we have a handful of really, really vocal students in our classrooms, and they participate a lot, and we have some really deep discussions with them, but it doesn't always create space for the rest of the class to get involved. And so there could be a little bit of that going on here, but we see a lot of back and forth between instructor and student.

>> Steven Robinow: A particular student.

>> Daniel Reinholz: And one particular student.

>> Steven Robinow: Right, so it's back and forth between an instructor and one particular student, so they're diving in, "So why do you think that?" You know, "What do you think? Why do you think it?"

>> Daniel Reinholz: Exactly.

>> Steven Robinow: Pursuing it with one student instead of incorporating many different students in that discussion.

>> Daniel Reinholz: And other students did get involved, but there was definitely much more back and forth with one student at a time.

>> Steven Robinow: Right.

>> Daniel Reinholz: Then sort of the distributed is really just kind of the middle. I'm not going to like focus on that as much. I don't think this is as telling. In the classrooms in the middle for the inequities that were like, relatively balanced between men and women, the contributions to participation was relatively balanced, and we saw ways that the instructor was using moves to sort of explicitly bring in other students. It could be like, they did a small group activity. "Okay, someone from group one, can you present? Okay, Jason, I saw that you had a different argument. Can you share that too? Okay, Emily, I know you had another idea, hear from you," so this sort of intentionality and bringing in different students to have more voices in the conversation. And then on sort of the other end was what we called shared inquiry, but what we noticed in some of these classrooms, well, they had the most participation from women, and we can't say that was causing performance, but there was something in those classrooms that was sort of supporting both of those things. We found there was a lot of students sort of leading the discussions and building on one another. It's hard to say, because we only had lesson -- we only had two lessons sampled from the semester, and nothing from like the very, very beginning, so I don't actually know exactly what the instructors had done in the beginning to set up that type of atmosphere, but they had built a classroom culture where students felt very comfortable to engage with one another, to build on one another, and sort of add to each other's contributions. That's a really profound environment when we think about inquiry where the instructor is providing guidance, but it's sort of just enough guidance that these -- the students can take over.

>> Steven Robinow: And those shared inquiry classes, women performed as well, or maybe slightly better than men.

>> Daniel Reinholz: Mm-hm, yup.

>> Steven Robinow: So super interesting, and those are in those classes also, the women had more participatory moments --

>> Daniel Reinholz: Yup.

>> Steven Robinow: -- than the other -- than both the distributed model and the centralized model.

>> Daniel Reinholz: Exactly.

>> Steven Robinow: So it's correlated with improvement, and now you're adding sort of a social aspect to it, which is that the class almost took the discussion as their responsibility so that students are talking and responding and reacting to one another, with some prompting from the instructor periodically, but not always, right?

>> Daniel Reinholz: Right.

>> Steven Robinow: Sometimes it's a back and forth between students.

>> Daniel Reinholz: Yeah.

>> Steven Robinow: That sounds like a pretty exciting classroom environment to be in.

>> Daniel Reinholz: Yeah, I'd like to be in that classroom. And so everything in this study is correlational, and we also know we're not looking at anything at the individualized student level. So like, I wouldn't want to like take one study and be like, we know everything, but what's useful is that actually, this study is very consistent with what we found in other research. Think about the earlier study that we did with EQUIP on gender. We saw the women were very successful in the small groups, but there were sort of this overall masculinized environment that was problematic and uninviting. Now, if we think about the shared inquiry classrooms, I can't say there was nothing that was unwelcoming or, you know, that was inequitable, but what we do know is the environment was comfortable enough that sort of all sorts of students from all different gender backgrounds, were actually just volunteering to contribute and volunteering to build on one another. So there's something in that classroom culture that actually is building that safety for students to be vulnerable and take chances when working on the disciplinary content, so I think that matters a lot.

>> Steven Robinow: Now I don't remember in the paper whether you said this, whether you talked about the gender of the instructors,

>> Daniel Reinholz: We did not talk about it too much in the article, because -- and this is interesting, because we looked at this in another study, and we actually didn't find any significant effects, and not to say that it doesn't matter, and I know that our own identities and positionality do matter in our classrooms, but we didn't find any specific correlations, at least in the studies that we did.

>> Steven Robinow: So the shared inquiry classrooms here, you had examples of male instructors and you had examples of female instructors.

>> Daniel Reinholz: Exactly.

>> Steven Robinow: And of course, as you say in your paper, you don't have the data, because the way it was run to talk about non-binary and so we're just talking binary here.

>> Daniel Reinholz: Right.

>> Steven Robinow: And in future studies, you'll -- I'm sure you'll investigate more deeply into those questions as well.

>> Daniel Reinholz: Yeah, we've done that in other studies. One of the things that's challenging is just for the quantitative work is overall sample size. When we do professional development with instructors, things like that come up in the data. For instance, it might be the EQUIP data part of the observation, where we're working with someone, and they've misgendered one of their students, but we noticed that. They didn't realize it in the moment, but we noticed that in the observation that we can sort of give that feedback and address that as something to think about moving forward, so we do see it come up. We do see gender nonconforming or queer students in the classrooms, and we know they have the different experiences, and we're able to unpack some of that sort of on the qualitative side, but if we have, you know, maybe one or two students in the classroom, then it's harder to do that with EQUIP.

>> Steven Robinow: Right. So you -- the sample size just becomes too small to get any statistically relevant data, yeah. So that's a situation where you'd have to move either to looking at thousands of upper division classes, or introductory courses that are much larger.

>> Daniel Reinholz: Yeah. And I just want to -- for one minute, I want to kind of come back to one thing with the shared inquiry, and that's to recognize in these classrooms the instructor was sort of in the background. We saw students building on each other. I think the take-home message is not as an instructor, just ask a question and hope for the best and hope that students build on each other, but actually there was -- it's evident there was a lot of intentionality, a lot of setup, building community agreements, those type of things that must have happened in these classrooms to allow for this sort of equitable exchange of ideas, because we've looked at hundreds of math classrooms, we've looked across other STEM disciplines, and we don't see that very often.

>> Steven Robinow: It sounds like you haven't done this yet. Have you gone back yet to talk to those instructors just out of interest to do a debrief and find out what is it that they do in their classrooms to make them such an open, engaging environment for their students? How do they do that?

>> Daniel Reinholz: No, we haven't done that yet. That would be an interesting study to do that.

>> Steven Robinow: I mean, it's a small -- obviously, it's a small number of people. It's probably -- in this study, it's probably four or five faculty --

>> Daniel Reinholz: Yeah.

>> Steven Robinow: -- but they must be doing some -- well, it would be interesting to know if there are common features --

>> Daniel Reinholz: Yup.

>> Steven Robinow: -- that they are using in their classrooms. And the faculty that you -- that taught these courses, they were run through a professional development program, specifically to run this Times program. Y

>> Daniel Reinholz: Yeah, yeah.

>> Steven Robinow: I'm sure they ranged in their overall experience in the classroom, some more junior some more senior.

>> Daniel Reinholz: Yeah, yeah. And just to be clear, the professional development was actually pretty extensive. The participants -- and now I didn't run it, so I'm not going to get all the details right, but the participants had some work in the summer, and then also, throughout the semester, or whenever they were implementing the curriculum, they actually had working groups, and they sort of had space to keep communicating on a regular basis with the team and other instructors to like sort of refine their practice, right? So this wasn't just unsupported. There actually was a fairly strong professional development model, but the thing the model didn't focus on, it didn't focus on the ways that gender inequity can come up or that racial inequity could come up, and so that gives me a little segue to some of the other work that we've been doing with EQUIP, right? -- because we have the data, and I think one of the things that we've been really successful at a field -- as a field at doing is documenting some of the challenges, and we know there's a lot of challenges. We know we have heterogeneous classrooms, and we know that race and gender play out in really subtle ways from stereotypes, positioning some students as competent, and others as not competent, or we know there are stereotype threats, that sort of, you know, for students who are from a minoritized background, they're put under this high-stress testing situation, and actually, those negative stereotypes reduce performance, because so much energy goes to managing the stereotype, and so there's a handful of things like this, that we know actually create barriers or actually can reproduce inequity, and so we know actually, creating an equitable classroom isn't always easy. And we also know that it takes intentionality, because if we don't specifically try to address some of these issues, then they're still going to be there. No matter how good our teaching is, no matter how active it is, no matter we have the best explanations or the best activities, we still have to contend with the fact that we live in a social world, we live in a world that is racist, sexist, there's a lot of other isms, and so unless we address those isms, they're going to happen in our classroom.

>> Steven Robinow: So as you're talking about addressing these issues in the classroom, I envision sort of two different things or many things that go on. I don't get the impression that you're talking about necessarily addressing the issues directly, like confronting the class and talking about it. It may have more to do with training the faculty in how to be culturally sensitive, culturally competent, other ways to raise these issues to help the faculty member be aware of the problems, and then provide tools that allow them to provide a learning environment that minimizes and hopefully obliterates these issues. You want to talk about that?

>> Daniel Reinholz: Yeah, I'd love to talk about some of the methods that we've been developing over about four to five years now. One of the things that we've found, and we're not the first ones to find this, others have noticed this before, but as a society, we tend to be a lot more comfortable talking about gender than we are talking about race, and some of that has shifted over time. There's a lot more conversations happening in the past few years, but we notice that with instructors too. To be clear, opening your classroom, letting someone in your classroom to videotape you or observe you, that's really vulnerable work, and so to do that, especially when the conversations are around, say racial equity, that just adds another level of vulnerability that we need to have a strong relationship. We needed to have trust. We need to respect the work that our teachers do both in K-12 schools, but also, college instructors. I genuinely believe just about everybody, 99.5% of instructors really care about their students, really want to bring the students into their discipline in a way that they can see the beauty and the joy for everybody. I think we have to come from that place that we're working together, and that this is hard. These are difficult things, and it's difficult to be an expert star researcher in your discipline, and then also learn everything about teaching that you can do and learn about culture and all these other things that you need to know to really make it happen. That's sort of the starting place. When we start from there, it gives us a foundation that we can build on. And so what we like to do, we sort of use EQUIP to ground our conversations, and that allows us to do a lot of different things in the classroom. So like the way our process works as an overview, we could set up a handful of observations, usually about three to four in a semester, and then we could do one on one, but I really like to get a group of faculty together. Let's get like three to five faculty, it could be the same discipline, could be different disciplines, and build a learning community, because folks have a lot of experience. They have a lot of great teaching strategies. They actually have usually been thinking about the issues themselves, to share that with one another, but what we do as facilitators is we observe. We provide observations between every time our community meets so that folks can get real data, concrete data about the patterns in their classrooms, and then what we can do is talk about strategies to address them, specific interventions that we might use, and then we try to do those, and then we measure again, see what happens, and then we can talk about it. So we have sort of this iterative improvement process. That's how we work with -- it could be faculty, it could be K-12 teachers, and what we found is that method is actually really effective at producing changes both in teaching, and also just in the ways that people think about either their work as a teacher or the ways that race, gender, disability, all these different identities play out in that classroom.

>> Steven Robinow: So this is a grant-funded program that you're doing, I assume, because it takes money to do this work, and you're reaching out to communities that are willing to participate in this. Have you considered or are you training an institution to do this themselves? Are you thinking about, you know, how do you -- I'm going to use this term. I don't know, if this is a good term to use. Have you considered weaponizing this, right? I mean, get so that you can, you know, release it at universities and let it -- let the faculty at universities work on this on their own?

>> Daniel Reinholz: Yeah, Steve. That's a great question, and so we've had a handful of studies sort of over the years to start building the methods that we're using, and I think for Niral and I, one of the things we wanted to be really certain about, was that it actually worked before we tried to get others to do it, because in our field, too often, we're just excited, we have some new thing, we want to scale it up as much as possible, but we don't necessarily have the evidence that it's actually as effective as we hoped for. So we've done the studies. We have a strong proof of concept, and right now, the grant funding that I have is through the NSF Career Program. I'm on the third year of that project and so for the first two years, I've been working with STEM faculty, first as participants in the coaching, but then I've actually been bringing back participants to serve as coaches, and so I just finished a year working with three faculty who were former participants who are now doing the coaching. And I wasn't sure how this would be. It's folks who don't necessarily have a formal background in education, don't necessarily have a formal background in coaching, but I was pleasantly surprised and really excited and encouraged by the work that folks I've been working with. They really took up the coaching methods. They've really actually helped make progress for the faculty they were working with, where they could make concrete changes to their teaching. Between their coaching sessions, we would come back together as a coaching group, talk about it, and develop strategies. I think you can see where I'm going with this. The idea here is to actually build the scalable model with waves of cohorts of faculty, who become faculty leaders working with other faculty and sort of passing this along, because we know also even as coaches, we're learning a lot of stuff. And most of what I've learned about teaching and equitable teaching is because I've both talked to so many faculty or I've observed hundreds of classrooms, so every time someone has a new trick, I'm like, okay, let's -- we can share that. And so I think for the same thing for the faculty, even when they start as participants and then later come in as coaches, the learning process just keeps continuing. And so hopefully, on some campuses, we can get on board to just pick this up because I really think what we found is that we actually do have potential to move the needle, but it does take time. It doesn't happen overnight, but we also know, I mean at least from the research literature, especially in math education, which is where I come from, when professional development is really effective, it tends to take a year. It often takes two years if we want to really start transforming practice, and so I think a lot of times our typical model is let's have a workshop Let's have a one-hour seminar. Not to say those things don't have value, they can spur new ideas, they can get people on board, but by itself, that one-hour session is not going to change practice.

>> Steven Robinow: Right. The one-offs are not sufficient, right? You need constant engagement, iterative engagement over time, yeah. So I really like your focus on improving things now. That's why I do this, because I think, you know, there are people who want to plan for the future, and that's great, but if we don't do things now, we'll never get there, so I think I really admire and applaud your work of working on things that are impacting classrooms in the very near future.

>> Daniel Reinholz: Thank you.

>> Steven Robinow: I really think that's great. So so far, you've been talking about STEM, this work in STEM courses. Do you have any idea or any evidence that this work is appropriate for non-STEM courses, it would be good?

>> Daniel Reinholz: I'll start with an anecdote. The EQUIP tool that I've been talking about is 100% free. If you go to the website, you can just pull it up in your browser, and then you can start using it, and so the tool is out there. And so it's slowly spread to people, either through word of mouth or through a podcast like this, or it could be other things. And so we had a school contact us a couple of years ago that just decided to use this for their coaching methods, and one of the things that came up in the discussion was how they were using EQUIP in their music classes to promote equity in music, and so we didn't have time to sort of dive into all the details, but it's just something that really has gotten me kind of interested from that point, right? Okay, this works in music. That's profound, and so I think the piece that we have is the customizability. The things that we code in a STEM classroom are probably going to look different from a music classroom, but sort of this overall methodology that we have, let's code something about participation. We combine that with student demographics, and then we can generate data in a way that supports learning. I think that is a really scalable model. Some of my own work has been primarily with STEM, but not only, and I actually found that when we had, like a community with some STEM faculty and some folks from the humanities or people from journalism or linguistics, actually, the cross-connections and the exchange of ideas was really rich. The way that we teach is different, but we can actually learn a lot from each other.

>> Steven Robinow: Okay, so you have engaged other disciplines --

>> Daniel Reinholz: Mm-hm, yeah.

>> Steven Robinow: -- and they are participating, and it's valuable. You see value there.

>> Daniel Reinholz: Yeah, we do.

>> Steven Robinow: That's great. That's cool. So let's disclose, you're excited about this notion of it being used in music, that they're about to find out that you're both a pianist and a drummer. You yourself are an avid musician, so.

>> Daniel Reinholz: Yeah.

>> Steven Robinow: Right?

>> Daniel Reinholz: Yeah.

>> Steven Robinow: So that's nice that a school came to you and talked about music. I'm sure that got you pretty excited.

>> Daniel Reinholz: I was very excited. Sometimes I call myself an accidental academic, because if I could have been a musician, that's probably what I would be right now.

>> Steven Robinow: Well, good to have a backup.

>> Daniel Reinholz: Yeah.

>> Steven Robinow: So let's spend a little time talking about implementation. When you talk to people who say, "Well, what -- how should I run my classroom?" what are some first steps that faculty can take first to determine whether they have gender issues, right? Do I have a gender issue in my classroom? And then if I do, what initial steps might you recommend?

>> Daniel Reinholz: Yeah, absolutely. So shameless teaser, I'm working on a couple of books, so in a couple of years, those will have a lot more information, but I'm happy to share some of the insights right here. So the first thing I just want to say is that we've probably analyzed 300, 400, maybe 500 classrooms. I've never observed a classroom that was perfectly equitable. It just hasn't happened. And, you know, this is even analyzing folks who have, you know, really senior instructors, even people who are really known, like well-known as educational researchers. It's just there in all of our classrooms to some extent. There's variation, but I've never seen something perfectly equitable, and to be honest, I'm not sure that we can measure what perfect equity would look like, but we're pretty good at measuring what's inequitable, and so we pretty much capture that. The second thing that's interesting is essentially every time that we've given the data to somebody, they're surprised. They might say like, oh, yeah, you know, I thought I had like a little bit of differences in gender. I didn't realize it so much or I didn't realize I was ignoring those students or just any number of things. It's usually a surprise. Folks, usually, once they see that data, they're like, okay, I'm ready to like, do something, right? You know, actually, this isn't that hard to fix, but I didn't have the awareness, so I couldn't address it before. Here's my top five quick interventions that we use. They're not perfect, but tend to shift practice a lot.

>> Steven Robinow: Okay.

>> Daniel Reinholz: One of the things that we'll observe in a lot of classrooms, especially college classrooms, is instructors just ask questions and students just shout out the answer. And sort of the rationale, right? -- we want to keep it casual. We want students to feel comfortable, and so forth. The problem is, whenever we do that, it's whichever students sort of feel the most comfortable or have the most privilege, and also, just white masculine culture is kind of competitive and kind of loud, and we talk over other folks, and so we see those patterns emerge every time the instructors just let students talk out. If that's your method, just having students raise their hands, and maybe not every time, but maybe you do it 75% of the time, that's already going to make a difference, but we can do better than that, right? We might have students raise their hand. We don't have to call on the first student who raises their hand, and if students shout out, we got to be like, no, no, no, hold on. I'm going to call on someone, just wait. Explain why you're doing it. Okay, I'm going to use hands, because we know some students might dominate in a discussion. When you get to participate, that helps their learning. I want to make sure everyone has a chance to learn. That's my job. I'm going to do my best to make it equitable, but rather than having just raise their hands, maybe say, I'm going to wait till I have three hands, and then I'm going to call on somebody. So we're already like sort of slowing down the discussion, and giving ourselves a little bit more latitude on bringing different voices in, so that's another very powerful strategy. Number three is think pair/share type of activity. Here's an example. When I was in math graduate school, I had all these classes. I said this is fascinating, I'm going to love it. And my instructors would ask a super hard question, and two seconds later, they would answer their own question, and we never had a chance to really think through it. You've probably been there in your own teaching where you asked -- I know I have -- you asked something, and it's just like crickets. Nobody's answering. Maybe it wasn't a good question, but it could also be a good question that students need more time to think about. In that situation, rather than answering your own question. Just break students off into pairs. Say I just want you to talk about this for a couple minutes to a partner, and then you can walk around the room and listen to those conversations, and if something was really unclear about your question, you're going to know it. If it wasn't, then you're giving students the time to dig into that deeper thing, and then you can come back as a whole class and get a response from someone. That's a helpful way to not just close off that action in your classroom. So that's three. Now I'm just on a roll.

>> Steven Robinow: You know, you don't have to do five. You can, if you like.

>> Daniel Reinholz: The last two are even better. So let's say that you're doing that, you have that turn and talk. Now, the way that I would like to do that is not just listen to students but, you know, we tend to know we might have some students who are quieter in our classroom or just aren't always getting the same opportunities. If I'm teaching a math classroom, I kind of know the inequities that generally arise, so I'm going to be aware of those and try to disrupt that, so I might pick a student that hasn't participated in a while and just kind of overhear what they're talking about, and just wait for one of those really good ideas to come out, and then I'm going to be like, hey, that's a really great idea. When we come back, do you mind sharing that? And usually framed that way, most students won't say no. It's very different from cold calling someone or a random calling where they're kind of stressed and scared. They're like, oh, I had a good idea, and you want me to share it with everybody. Great, I'll do that. And that's actually a way to very intentionally both bring voices that might not be heard, and also elevate their status in the classroom, right? -- so they can actually feel some more of that belonging, feel more of that success. We can stop there. That's number four, but I think just that very small progression of strategies for facilitating discussion has a profound impact on the types of patterns that we see in who's participating and how.

>> Steven Robinow: Right. One thing that I really like about that is the issue of time where we so often -- we end up with the first person to raise their hand or the first person to shout it out. I mean, so much of what we do is time based. You forget that the class has a certain amount of time. Of course it does, but exams have a certain amount of time, so and how we test, we select for certain things when we do that. And so time in the classroom excludes so many people. Some people might just need more time to get there, but they'll -- but give them the time and they will, and then --

>> Daniel Reinholz: Exactly.

>> Steven Robinow: -- if you give them the time, then they get to participate, they get to feel like they belong, and every time you cut them off, and don't let them get there, they just check out because, you know, you haven't given them the time that they need. So I really like that notion of changing the dynamics of the time --

>> Daniel Reinholz: Yup.

>> Steven Robinow: -- and you've given lots of tools that then how to use that and to bring other people into the conversation. That's cool. Before we move on --

>> Daniel Reinholz: Sure.

>> Steven Robinow: -- to sort of maybe the final part, is there anything that we didn't talk about that you want to?

>> Daniel Reinholz: I'll just summarize. I think the key thing that we find, when we have EQUIP, when we have these strategies, is recognizing that a lot of the patterns that we see, students aren't just shy. We don't just have lazy students. Like, I think it's easy to blame stuff on students, but it's actually the patterns that we find are very predictable. They're very strongly connected to these social marker identities like race and gender and disability. It just puts a little bit of onus on us to when we recognize that these are systemic problems, there's actually a lot we can do without a lot of work. Those four strategies I gave you, almost any faculty member that I've worked with, by the end of a semester, they can implement most of those pretty effectively. The first time you try to do it, it might not go great. The second time might be okay, but you try it a few times, and you get the hang of it pretty quickly.

>> Steven Robinow: Yeah, and also, what that's reminding me of is, as you were saying that, it's just students being seen, right? -- having the faculty recognize them and recognizing them and valuing them for what they have to say. The quiet students want to be recognized as much as the non-quiet students. There's a lot of relational value that's generated there when you -- when students understand that you see them.

>> Daniel Reinholz: Exactly, and it doesn't take a lot. Well, we have to learn our students names to begin. I mean, that's just sort of the basic piece but, you know, just talking to our students, spending five minutes before our class starts just to get to know them a little bit better, I think goes a really long way.

>> Steven Robinow: Absolutely, absolutely. I agree. So I want to close with -- I don't know if this is going to be one or two questions. The first one, I guess I'd like to know when and why you became interested in equity and disability issues in the academy. And I'm wondering, did you bring these concerns with you when you became an academic? Or did your experiences in higher ed sort of force these issues upon you?

>> Daniel Reinholz: I think it's a mix. So like I mentioned earlier, I'm sort of an accidental academic. Music was sort of my first passion. As a teenager, I got into a lot of trouble. I was really into punk rock, and really had that social justice bent, but like also like, sort of in a troublemakery kind of way. I think it really is music that gave me that first push towards social justice, and that's where I come from, and you know, I've learned a lot from others and brought on more nuanced perspectives that I didn't have before, but that was kind of what started it. And it was interesting, because when I started, you know, I started in engineering. I switched over to math, and like, I loved the science, but it was a lot more technical. Even in my doctorate, the research I did was interesting, but kind of traditional. I was looking at how -- what happens when students critique ideas of their peers, and like, can we give feedback to our peers to learn from it? So it was cool. It turns out, yeah, we can actually improve learning by doing some of those peer assessment types of things. Somewhere along the way, I started to see that I could take on those equity and social justice issues, and it had a place in the academy. It had a place in my field. And actually, in math education, that has sort of become a huge focus. Especially in the last few years, I kind of started that doing that work a little bit before it was the big thing to be doing, but now it's sort of fit into that place with what I'm doing. So that's kind of kind of what got me into equity work. In terms of the disability stuff, that's been a new piece for me in my research, and really, that comes just from my own experiences as a disabled person. I've had chronic health and autoimmune issues since a very young age. There were a lot of barriers in college in the engineering programs and other things that I didn't even realize were problematic, because they were just so normalized. I just kind of accepted them and everything was my fault. I just was like, if I can just work harder and just work more, it'll be fine, I'll get through it. And so it's really been -- and I think, especially with a COVID pandemic, and seeing how that's really impacted disabled people disproportionately affected racially minoritized folks, low-income folks, but also, this is a mass-disabling event. The number of people that have died or that have long COVID now are actually having that new experience of perhaps going back to work, but actually being disabled now and trying to manage that. I think experiencing sort of that pressing problem, having gotten COVID twice, myself, which was pretty scary the first time --

>> Steven Robinow: Yeah.

>> Daniel Reinholz: -- and scary, but less scary the second time, just hit home that I could do this work, and that there's actually not very many people in our fields doing that. To take a positive spin, you know, in the last few years, I've really been getting connected with other folks in the disability community, and to have sort of that mutual understanding and awareness and support has been very profound. It's been very transformative for my own life. We look for ways to pay that forward and kind of build that in our community, and so one project that I'll just give a throw out to, and so this was a collaboration with Lisette Torres-Gerald. We're building a community science of disability to try to actually build recognition and support for disabled mathematicians. You know, if I asked you to think of your top five disabled mathematicians, I wonder how many you can think of, and that might be similar for other fields other than kind of like canonical examples that people bring to mind. That's an effort to sort of build out that community, and it's also actually influenced the disability work into some of the EQUIP work, and so I mentioned earlier, some of our work has actually focused beyond the verbal, but all the different modes of participation, and so this is an ongoing conversation with my colleague, Katherine Ye, where she's been like, you got to capture that. And so we've actually started building out a project working with teachers, and that's been really exciting too.

>> Steven Robinow: Very cool. I wonder if you've spoken to Jeff Schinske about his scientist spotlight program. It sounds like that's something worth looking at and thinking about.

>> Daniel Reinholz: I haven't, but I'd love to do that.

>> Steven Robinow: Yeah, that's a -- well, there's an episode with Jeff Schinske, and you can listen to it and I can put you in touch. Last question, last question. You talked about getting involved in social justice through your music. I wonder is there a particular story you want to tell, because telling me that you were a punk rocker and you cared about social justice, I wasn't a punk rocker, so I don't know that connection? I don't know that all punk rockers are into social justice. Maybe they are. I hope so, but is there a particular story that sort of that you think about?

>> Daniel Reinholz: Well, most of my stories are kind of getting into trouble stories.

>> Steven Robinow: That's okay. It's okay, if you want to admit it here. It's okay, but you don't have to.

>> Daniel Reinholz: Right. But I mean, I think the connection between punk rock and social justice, there's sort of this existential angst, but also this need for change in like the political system and the way that we work out in society, and so I think about just a lot of those punk rock musicians, you know, Bad Religion, Anti Flag, Rise Against, Pennywise, many, many more I can name, sort of had those messages, and they really resonated with me as a teenager, especially because that was the time when I started getting really ill, and sort of started not knowing how to navigate that feeling, I think both this sort of internal sense of injustice, but also a sense of injustice in the world, and so I think it sort of struck the right chord with me at a really hard time in my life. Ever since then, I like believed in the transformative power of music. That's what's made me a punk rocker for life.

>> Steven Robinow: All right. So we're going to put some of those names up on the website.

>> Daniel Reinholz: Okay.

>> Steven Robinow: We'll put a link for maybe your favorite music or your most impactful bands.

>> Daniel Reinholz: Great.

>> Steven Robinow: Daniel, I want to thank you so much for your time. I've really enjoyed our conversation. This has been a really fascinating discussion. You've given me and hopefully our listeners important issues to consider, and hopefully some tools to address some of these issues. Thank you for your time. Really appreciate it.

>> Daniel Reinholz: Yeah. Likewise, Steve. I've really enjoyed the conversation, and again, I'm very grateful for the chance to speak with you and speak to your listeners, so thanks so much for that.

>> Steven Robinow: Our website teachingforstudentsuccess.org will have additional information about Dr. Reinholz and their research on equity analytics. We will also include information about their favorite punk rock bands. Thank you for spending time with us today. Please share our podcast and website with your friends. Please rate us on your favorite podcast app. We're trying to increase listenership. This would be extremely helpful. Thanks so much. Those of us at Teaching for Student Success would love your feedback. Please contact us through our website at teachingforstudentsuccess.org. Teaching for Student Success is a production of Teaching for Student Success Media. Let's end this podcast with some music by Julius H. Some of Julius's music can be found on Pixabay. And maybe, Daniel, I should ask if you would be interested in composing some music for us.