Panel 2 Speaker: Amanda VanDerHeyden, Ph.D. - Consultant (AL)
RTI Leadership Forum
December 8, 2010
You know what? You’re not going to be able to see me. I just realized that is so unfair. Can I step down now? (laughs) He asked me will I stand up? I am standing up (laughs). All right. I promise not to sing. And I promise not to trip, I hope. Okay. My strategy today…first of all thank you for inviting me to do this. My favorite group to talk to are implementers, people who are on the front lines and I think we’re really at an implementation crossroads and it’s exciting because I think the things that are easy to train and easy to set up are underway in many places. Matt Burns and George Noell published a really fantastic study in the Journal of School Psychology and they looked at what components of RTI could just sort of easily be gotten into place with some up-front training and some what’s called performance feedback, which I know you all are familiar with. And what was interesting to me about that study is the things that they had…the behaviors that they had success with were things like holding a weekly meeting. Right? You can probably guess what did they have the most kind of trouble with. Go ahead, choral respond.
(audience member speaks – mic doesn’t pick it up)
Yeah. The stuff we struggle with. Adults being responsive to the data. So my strategy today was to be just a little bit provocative, and so I want to make an analogy to prostate cancer and screening for prostate cancer and I thought well at least I’ll have your attention (laughs). So actually I think it’s pretty relevant to us and I think we can adopt some, we can learn something from this strategy that Jerry Hoffman and colleagues have developed that can push us forward to get better results for students and more importantly keep our eye on the, is it bear and bushel or bushel and bear—the close and continual contact with the outcomes that matter, right? I can say as a parent if I have faith that someone is being responsive to the data, then most of my fears related to which data are being collected start to dissipate because I have faith that something is being done. I think in many places, nothing is being done. So that’s what I’m going to talk about today. In fact, oh, I can’t advance from here. Can you do it for me?
In fact, I’ll start with a personal anecdote. I think all good researchers, not that, I guess I think I’m a good researcher, all researchers should have a child and send them off to school. That is a wonderful thing and I have a son in first grade and I learned this year that at the beginning of the year that the school library would be closed for three weeks so that the librarian and other professionals in the school could conduct all the reading assessments of our children. And I thought, oh no! I mean for years people like me have said we need to collect data, we must do these assessments, this is important. And yet are we missing the forest for the trees when we are closing the library for 10% of the school year in the name of advancing literacy? And I don’t know what’s going to happen at winter screening and spring screening. They may close that library again, and if they do, I will have a heart attack because it makes no sense and so my passion as a researcher is focusing on things that we can do that make a difference. I want to do things that actually make a difference, and I think if we hold ourselves to that standard then a lot of the controversies simply go away because they don’t matter. What we call a kid doesn’t necessarily matter. If we can demonstrate that as a result of spending this money in this way to do this assessment, to do this intervention, increases the odds of that child being a successful reader or going to college or not dropping out of school. That’s the stuff that matters. So I want to show you a way to do that with prostate cancer being an example.
So first of all we have to make multiple decisions in RTI. I like to think of this as multiple chances or opportunities to have error in the process, right? Decisions are the sticking point for many, many schools. You can advance to the next slide, please. Ut-oh. Got it? No? Shouldn’t be. Okay.
And this is sort of how I feel, and I feel this way even about my son’s school that in fact schools are drowning in data, right?, and if you ask people to explain well why did you collect this data point, how do you intend to use it, and what are you going to do differently tomorrow as a result of having collected this data point, people cannot give you a good answer in many cases. So I think we need a framework for evaluating which actions, which assessments, which decision rules, which interventions, and which intervention decisions rules are helping us reach our target of improved student learning outcomes. Go ahead. It’s probably because it’s a Mac that I created it on.
And so here is the framework that I stole from Jerry Hoffman who is a great emergency medicine researcher at UCLA, and he developed this framework called, this aid to accomplish what he refers to and people in the medical community refer to as “shared decision making.” And he suggested that this is what you could share with patients so that patients could really become partners in their medical decision making about which tests to endure, right? There is no mercy in an urologist’s office, right? But so that patients could have a function in that, have a place in that decision that is meaningful. I think actually, actually, and I emailed him this and shared this, and he said he really does agree, that doctors need this decision framework because they also do not have a good grasp of this framework for decision making. Here’s what it looks like. Stolen from Jerry Hoffman and colleagues. And by the way the citations at the bottom, and you can get it online, it’s available for free, open access, and it is a fantastic little article. My husband is an ER physician and I can’t get him to read it, but hopefully he’ll read it so….at any rate, what you’re looking at here if you’re trying to make a decision to agree to have prostate screening or not, in this category for a 65-year-old male the chances of being asymptomatic in the following several years are represented by the green area, and the chances of death are red, represented by red, and the chances of negative side effects that might occur because of false positive errors that you look like you’ve got a prostate cancer so then they go in and do a surgery, and by the way, anytime you have surgery there’s a chance that something bad will happen. Just like in what we do. If you go in and provide unnecessary intervention it is a zero-sum game and it comes at a cost to other things. So that is represented by the yellow. And so then the question is gee, if you’re 65 years old, would you choose to have a prostate screening? See we could use the clicker for that. You could respond. It would be interesting to see right, because actually chances of death do decrease, but only slightly, right, because actually chances of death are not that high to start with. You might just want to take your odds and roll the dice because the chances of having negative side effects resulting from unnecessary treatment increase substantially.
Okay so he begins with the pie chart and then he says, okay, well let’s…sorry, advance for me. Let’s extend this framework to a dart-board concept, and this will more evenly show you what your chances would be. So in this case red again is chances of death, yellow is risk of negative side effects, and the green is being asymptomatic. Which board do you want to throw a dart at? Okay? He actually extends this idea a little further and says, Well, you know a dart board arguably, you could influence where your dart goes based on skill and so he says really ideally it would be a roulette wheel because then it truly is a game of probability. And so then the question is if you look at this, would you choose to have a screening or not? So imagine what that might look like if we were talking about reading screening. Reading screening at certain ages. Sometimes we make decisions about collecting screening data when it would be simply smarter for us to intervene. Now by the way as a patient usually you’re not convinced by these type of data, right? So if you’re told this treatment works for only 10% of people who get it and you feel badly enough and you want that treatment, you don’t care, right? And the other thing you want to know is well, wait a minute, am I going to be in the red? Or am I going to be in the yellow before I decide which one I want to aim at? But from a systems perspective, from a systems perspective this idea of looking at probability of outcomes and changes in outcomes that can result from different procedures and different interventions can help us allocate resources in ways that we get better results over all. And that’s what it’s all about anyway. So can you advance one more slide? I think I have one more there. Yup.
Okay, now let’s talk about treatment because now it gets more complicated. And so think about in a high-achieving low-achieving school, the amount of red on the dart board is going to be totally different, right? And so in some cases here you’re looking at again with the prostate cancer example, this is the chances of death, or red, right?, with no treatment. So this is you elect not to have treatment, your chances of death are substantially higher than here, right? If you elect to have treatment, you do have now some chances of negative side effects, but chances of death go way, way down. That makes this a pretty easy decision, right? But the things you want to know are what are the possible side effects and how bad are they of unnecessary treatment, right, as a patient. And what are my chances of dying if I do nothing? Well, we could say the same thing with kids in RTI. What are the chances this child will not learn to read? What are the chances that this child will fail math? And the amount of red that’s on the board, failure in math, dropping out of school, would look different based on certain contexts, and that is quantifiable. And yet we do a terrible job of it, and Meehl and Rosen said it all in 1955, that prevalence should guide some of the decisions that we make. I made this comment in Wisconsin and totally inspired a visceral reaction from a woman in the audience. I said You know in some cases we probably could make a more efficient and accurate judgment to simply begin intervention based on whether or not a child receives free or reduced lunch, and how old they are. Or, similarly, we could just decide to begin intervention in a school that we know is very, very low achieving. And the reason we might want to do that is because there’s probably more error associated with the screening and the decision rules than we would get if we simply assumed everybody had a problem and gave them all intervention, okay?
And that’s going to sound like it’s at odds with my last slide, which is the concept of more is not better, and I see this all the time in schools. Again my son, perfect example. The idea is more assessment must be better. No. There are consequences associated with all of the decisions that we make, and I think as RTI implementation people, RTI implementation leaders, we have not done a great job of getting our arms around what are the consequences of unnecessary treatment? What does that come at a cost to? Cause I can tell you, working in schools they’re taking it from science class, they’re taking it from recreational opportunity, they’re taking it from field trips. It’s not benign. I would love for my child to be sure that he gets to go to art class and PE have a nice recess and make a puppet for a puppet show on Friday, you know? I would like for him to be able to do that. I happen to believe he can do that and still learn to read if we are more efficient about the decisions we make. I just worked with a kindergarten center and calculated for those teachers, this is the amount of time you spend in assessment at the kindergarten center. 11% of their instructional time is devoted to assessment. Is that a good use of time? I don’t know. You know what my question is? Does it change the odds? Does it change the amount of red on the dart board? Does it actually have a measurable effect, and we can quantify that. That’s what we should be looking at in terms of screening scores for decision rules. We should be looking at if we apply this decision rule, if we conduct this assessment, odds of a child learning to read successfully at grade 3 go up, better than chance. Or we shouldn’t be doing it because it comes from instructional opportunity and opportunities for other rest and recreation.
Okay, so this means for us we need to move away from burdensome screening rules that give us too many false positive errors. We need to have that shift in thinking that more intervention and more assessment is not necessarily better. I’m echoing what people have said earlier on the panel which is we need, as adults, as implementers, we need to be more responsive. We need to have close and continual contact with the implementation outcomes that matter, and where we’re not getting results we need to be responsive to that. And we need to use assessment strategies that can be shown to change the post-test odds of child success. Those are easy to calculate, but I think, publishers in the room, we should have technical innovations that do that for us behind the scenes. You don’t have to understand it. You don’t even have to get your calculator out, and you can get your dart board and help parents look at that and make a decision about whether to begin intervention and what type of intervention to begin.
And finally I think I’ve glossed over this entirely, but we can prevent implementation failures by reducing operator error and in other fields we see this a lot. I know in my car my cruise control will automatically adjust if there’s a car in front of me and I get a little light that tells me that my car needs maintenance. We could embed some of this technology into RTI implementation to get better results, and it’s doable. It’s viable with some of our web-based software management. And that’s it. I finished on time, huh? (laughs) (applause)
Back To Top