Colorado’s RTI Implementation Rubrics

Adena Miller, M.A.
Start with your data. Consider the questions in the problem-solving rubric: How is the problem-solving process used by educators and families to improve outcomes for groups of students? Have you looked at your universal data? Are you effectively meeting 80% of your students’ needs in general education? If not, why not? What could you be doing differently to ensure you are? How is the problem-solving process used by educators and families to improve outcomes for targeted groups of students? Have you identified common areas of need for those students? What interventions could address those areas of need? For those individual students for whom targeted interventions don’t work, what data do you have? What are their needs? Use your data to inform what (if any) new interventions are needed. Before seeking out anything new, also collect data on what your school already has in place. Is there a closet where interventions are hiding? Does a teacher have a skill set nobody knows about? Once you have used your data to determine what you need, send out an all-call to see if anyone in your building can address it.

Adena Miller, M.A.
SLPs roles vary depending on the school and district they are working in. In some cases they are included on the team that discusses RtI implementation and the rubrics, but frequently they are unable to because their case loads are so large and they are assigned to so many different schools. My perception is that the most common role for SLPs is to participate on individual student problem-solving teams. Certainly they have a lot to contribute, unfortunately, it is just sometimes difficult to make the scheduling work out.

Adena Miller, M.A.
Start with your data. Look at the rubrics—what do you have in place? What is missing? What are your priorities? For many, RtI implementation is not about buying new programs, or more interventions, or hiring new teachers. It’s about re-thinking what is in place. Consider the questions in the problem-solving rubric: How is the problem-solving process used by educators and families to improve outcomes for groups of students? Have you looked at your universal data? Are you effectively meeting 80% of your students’ needs in general education? If not, why not? What could you be doing differently to ensure you are? Many people at the developing stage realize that they have not adequately addressed universal instruction. They start trying to find interventions for all of the students who are not working at grade level and find that there aren’t enough teachers or resources to address those needs. At that point they revisit universal and make changes to the infrastructure there. I’m not sure if that is a challenge at your school, but I would encourage you to always start with the data when making any decisions. What does it tell you? What is the biggest priority or challenge? What is your hypothesis? Can you verify it? If so, what actions might remedy the challenge(s)? If you don’t have problem-solving processes in place for the universal, targeted, and intensive levels, this might be the thing that gets you the biggest bang for the buck!

Adena Miller, M.A.
Before moving to action, I would encourage your school to review the rubrics and determine you stage of implementation for each of the six components. You may be surprised by what you already have in place. Additionally, it is imperative that you work to develop consensus first. This is the stage that many schools skip, and if you don’t have this first, you will struggle to implement at scale. As you review the rubrics, consider what the school’s priorities are and what the smallest changes you can make are that will give you the biggest bang for the buck. Take the time to develop infrastructures so that you set yourselves up for success when you finally get to the operationalizing stage.

Adena Miller, M.A.
The Implementation Rubrics center around the 6 components of RtI that Colorado has identified. These include Leadership, Curriculum & Instruction, Assessment, Problem-Solving, Positive School Climate & Culture, and Family & Community Partnering. If those other initiatives (such as core curriculum) align with any of those components, I would encourage you to look at the guiding questions and stages of implementation to consider how the tool may help you plan your reform efforts. Additionally, consider how RtI really lends itself to providing a framework for how all of those efforts are intertwined. I would hope they are all interrelated in some way, as the RtI components are, to create an ensemble. As Peter Senge pointed out “this is challenging because it is much harder to integrate new tools than to simply apply them separately” (Senge, 1990, p, 12). Fusing the efforts keeps them from becoming the latest fad, or individual gimmicks.

Adena Miller, M.A.
In the spring of 2010, Daphne Pereles, Executive Director of the Support & Intervention Unit at the Colorado Department of Education (CDE) hired Dr. Mary Ruth Coleman from the University of North Carolina to facilitate the development of the rubrics. Dr. Coleman and Daphne Pereles developed guidelines for rubric development. Members from a cross-unit team at the Colorado Department of Education worked together with Dr. Coleman to develop growth stage based on the work of the State Implementation & Scaling-up of Evidence-based Practices (SISEP) Center (Fixsen & Blase) and were assigned to writing teams. The cross unit team working on the rubrics included representation from the following offices and units: RtI, Teaching & Learning, Exceptional Student Services, Title I, Gifted & Talented, Early Childhood. And Language, Culture and Equity Representatives from a few school districts implementing RtI effectively also participated in the process. Prior to the actual development of the rubrics, the cross unit team traveled across the state to ask questions of school and district personnel regarding the use of the rubrics. We used a standard protocol approach with focus groups and asked for input on how these tool could be created in such a way as to support schools, create meaningful dialogue, and reflection on practices. This information was utilized as the writing teams drafted the rubrics for each component. Once the writing was complete, we hosted a symposium with teams representing 8 districts to gather feedback on the draft rubrics. From that point, we refined, edited, and published and trained on them. The 2011-2012 school year was the first year that schools utilized these tools. We have just completed gathering spring data from those systems.

To answer your question regarding online tools—I don’t think there will be one right way. At this point, we do not have the resources to program this. We have developed a Survey Monkey Tool to collect data from the mini-grant districts, but it only contains the guiding questions and the ability to select growth stages. Ideally we’d love to have this available online, but I don’t foresee that in the near future. Regardless, I think it is critical to have dialogue and conversations among school staffs before determining a growth stage. Creating an electronic tool would allow for better monitoring over time, but I would hate to see it detract from the process.

Adena Miller, M.A.
While there is no one right team for conducting the review, we do have recommendations. It is important that a building administrator participate in the team. Additionally, you will want team members with different perspectives (e.g. general ed, special ed, different content areas or grade levels, school psychologists, counselors, etc.). If the school has an existing leadership team in place (such as department chairs) you may consider using this team as well. Some schools have elected to discuss the rubric as a complete staff before rating themselves. If they don’t do this, the leadership team may plan together how best to share their perceptions with the remainder of the school staff. The most critical role for the rubrics to play is to encourage reflection on practice, generate dialogue about what is effective, and action plan around prioritized areas for growth. If the school leadership wants the full staff to be able to move forward, they will need to have consensus among those educators that the proposed plan of action is the best way to do so. Skipping that consensus building stage is often the biggest barrier to implementing an initiative such as RtI.

Adena Miller, M.A.
As described in the previous question, the most critical role for the rubrics to play is to encourage reflection on practice, generate dialogue about what is effective, and action plan around prioritized areas for growth. If the school leadership wants the full staff to be able to move forward, then they will need to have consensus among those educators that the proposed plan of action is the best way to do so. Skipping that consensus building stage is often the biggest barrier to implementing an initiative such as RtI. The rubrics provide a nice vehicle for discussion, reflection, and decision-making. It is important to remember that although this is a qualitative rubric, decisions should be based on various data sources that are triangulated to verify perceptions. This can include student assessment data; walk through data; staff, student and family satisfaction surveys; interviews of staff members, family members, or students; meeting minutes; policy and procedure handbooks, just to name a few. When staff have an opportunity reflect objectively on data and how processes and procedures are working the school, they begin to see opportunities for growth and improvement. Ensuring teachers have a foundation in understanding the “why” behind RtI and have a voice in planning can make a big difference for buy-in.

Adena Miller, M.A.
CDE developed a graphic (which can be seen below and on page 11 of our RtI Implementation Rubrics Guidebook) to reflect how the rubrics can support a systemic approach to RtI implementation across the district: Classroom level rubrics and data analysis will feed into the school level, which will then feed into the district level rubrics. Districts can then strategize and develop a plan to support schools with their implementation based on need, which will guide the school level, and the school will guide the classroom work. As all of these levels implement RtI and their action plans, more data will be collected and analyzed as part of the iterative process. At the center of it all, the ultimate target, is student achievement. We believe that the data gathered through the school level rubrics will really support districts in knowing which areas to target their professional development and support.

Adena Miller, M.A.
The Implementation Rubrics are written as blueprints or roadmaps of RtI. Because RtI is building specific, they were written from a descriptive standpoint and not a prescriptive one. For example, assessment is a component of RtI, so each site that implements RtI will utilize regular assessments and screeners as features of their implementation. However, like snowflakes, no two RtI models will be identical because of situational factors (e.g., resources, personnel, size of student population, etc.). Therefore, the Implementation Rubrics only say that screeners and regular assessment should be used, but does not specify which tools to use, how often, or by whom they should be administered. The Implementation Rubrics outline what the components of RtI look like without describing the model in such detail that it sacrifices the flexible nature of RtI (Excerpted from RtI Implementation Rubrics Guidebook , 2010, pg. 7). Like the rubrics, districts can do a lot to facilitate the RtI process, while still allowing local flexibility at the building level. Many of the structures and processes will be similar, but the specifics of how or what will be implemented may vary.

Dan Jorgensen, M.A.
It’s important to emphasize that the rubrics were designed to support understanding and implementation of the RtI framework and not to prescribe practices. In effect, the utilization of the rubrics has differed depending on the specific needs of the sites. We’ve had reports of some districts focusing on individual rubric components and monitoring change during the year. Other districts have focused on the totality of the rubric and use it to facilitate discussion regarding their implementation process. The feedback that we’ve received via an RtI Action Network survey has been very favorable. The majority of survey respondents view the rubrics as being beneficial for implementation, as being comprehensive enough, capturing implementation successes and serving to improve RtI program coordination. This feedback coincides with what we’ve been hearing from practitioners within Colorado.

Dan Jorgensen, M.A.
In answering this question, it’s important to recognize that the primary purpose of the rubrics is to facilitate improved understanding of the Colorado RtI model. The objective isn’t to rank order or prescribe a course of action for districts/sites so the rubrics were purposefully developed to be qualitative in nature. In effect, the rubrics were designed to improve implementation and depth of understanding of the Colorado Response to Intervention model. We’re currently involved in the collection of pilot data from a number of sites across the state. To date, our quantitative analysis has been limited to an examination of where sites are finding themselves in terms of growth stage for each of the rubric components. To present, 61 sites have provided us with baseline growth stage data on all six of the rubric components. It appears based on this pilot data that a future point of emphasis may be in regards to family and community partnering as 79% of the sites self-reported as being at the emerging or developing stage of implementation. In contrast, for the leadership component only 57% of sites self-reported at these levels. Future work on the rubrics will include a series of studies that begin to examine the psychometric properties of the rubrics. Again, this hasn’t been a point of emphasis due to our focus on the conceptual value of the rubrics. It’s worth mentioning that we’ve recently initiated work on the development of a tool to complement our “qualitative” rubrics that will be based on more observable indicators of implementation. This tool will be aligned to our six RtI components and will provide us with a standardized measure that provides overall scores, scale scores, fidelity cuts, and is psychometrically sound. Our goal is to pilot this tool later this year. I’m hoping to be able to share the tool and technical report at some point next year.

Dan Jorgensen, M.A.
For a complete list of resources please visit the Colorado Deprartment of Education RtI Tools and Resources page. Also, please see the documents posted on the RTI Action Network website. If anyone has specific questions about the rubric in the future feel free to e-mail Dan Jorgensen at: jorgensen_d@cde.state.co.us

Adena Miller, M.A.
When we designed the rubrics, we included our preschool team in the process. We are hoping they are applicable to the preschool level, but have not heard direct feeback on this yet.

Adena Miller, M.A.
Our rubrics contain some guidance regarding including families in the process. Another great place for information is our Family & Community Involvement webpage. We have developed a toolkit which can be found there with a lot of ideas, tools, and resources for including families across the tiers. There is far more information there than I can capture here, so I would recommend going directly to the toolkit. It contains sample letters, brochures, Power Points, etc. that are all in PDF as well as Word/PPT so that you can take them and customize them for your school or district.

Dan Jorgensen, M.A.
This is a great question. It's important to recognize that growth is site specific and is not necessarily associated with specific timelines. It's not uncommon for large differences to exist in regards to different components and time to implement. Also, growth isn't always linear. It's quite possible that growth will be mitigated or even reversed by systemic and/or unforseen changes in leadership and/or capacity. The key is to keep an eye on a continuous improvement perspective

Dan Jorgensen, M.A.
Yes, effective program implementation of any type is predicated on buy-in and understanding of the proposed framework. If possible, I would recommend that all involved staff and leadership team members receive such training. For the rubrics we've developed a PowerPoint to assist in this regard that's located at the Colorado Department of Education RtI Tools and Resources page. 

Adena Miller, M.A.
This is an exerpt from page 14 of our RtI Implementation Rubrics Guidebook:

Fidelity is the extent to which an approach or intervention is used as it was intended, defined, or designed (Coleman, Shah-Coletrane, 2010; Lane, Bocian, MacMillan & Gresham, 2004; Gresham, 2004). Approaches or interventions are “practices” we use in the education setting, and they range in scope from classroom strategies to full systems approaches (such as RtI and PBIS). To consider whether a practice has been implemented with fidelity, the practice must first be clearly defined (Century, Rudnick & Freeman, 2010). The definition of the practice should provide a comprehensive description of what the practice will look like when it is implemented as intended, and it should address explicit expectations for implementation. By clearly defining a practice, expectations are spelled out creating an understanding of what needs to be accomplished. From that understanding, educators are able to reflect on the integrity of their work and then plan for next steps to improve implementation (Fixsen, Blasé, Horner & Sugai, 2009). In most cases, a rubric is used to guide reflections on practice and to document levels of implementation fidelity (Century, Rudnick & Freeman, 2010). The Implementation Rubrics were created for Colorado’s implementation of RtI.

Adena Miller, M.A.
This varies from site to site, and depends on the data and systems you have in place. For the schools our RtI Technical Assistance Coordinators (TACs) work with who are newer to RtI, we typically do a training to understand their intent and purpose (this can be accessed on our webpage here: http://www.cde.state.co.us/media/RTI/RTIRubricTraining/player.html), and then allow for at least 1-2 hours for conversation. Many school staff will go back, collect and data (including conversations with their peers, students and family members), and then regather to finalize their rubric results. Overall, in meeting time, I would not aniticipate more than a half day, however, if you already have data in one place, and systems to reflect on practices, it could take much less time.

Dan Jorgensen, M.A.
Please see Adena's response to Ann above. It's important to recognize that effective family-community partnerships are invaluable to having a successful RtI model. My work has focused on providing districts, sites, and the Department of Education data regarding progress in this domain. The rubric data is key and has shown us that we have more work to do in this regard. In addition, we've collected survey data from approximately 5,000 parents/guardians across the state this year to better understand the performance of individual sites and districts in developing these relationships (i.e. regarding both academics and behavior). This information is provided to the field to help support them in their practices. For many of the smaller districts and sites this has been invaluable given limitations on internal capacity.

Adena Miller, M.A.
I think it's important to understand what a school already has in place, and whether the school has discussed and/or begun RtI implementation. If a school has not started RtI implementation, it is critical to go through the Emerging stage —in this stage the school looks at what is in place and what the gaps are; they seek to understand RtI and WHY they might want to adopt an RtI framework. They develop consensus (generally at least 80% of the staff on board) and prioritize next steps. Our rubrics provide guiding questions that can help facilitate this process. After consensus has been established, the school will move into a developing stage where they start to develop infrastructures. These infrastructures may support any of the 6 components (Leadership, Problem-Solving, Curriculum & Instruction, Assessment, Positive Climate & Culture, Family & Community Partnering), and prioritization occurs at the school level. Once infrastructures have been built, a school moves into the operationalizing stage—this is when you put in place that which you planned. Keep in mind that the stages are not linear, and that you can be in more than one growth stage at once. For example, you may plan some interventions for literacy and then realize you have far more students who need the intervention than you have interventionists. At this point you may go back to the developing stage to redesign universal instruction so that you are meeting more students needs, and need fewer interventionists. The last stage is optimizing —this is when all practices are embedded, and part of the way you do business. This is not the end of implementation, however. At this stage the staff continuously problem solves, tweaks and adjust practices and processes to refine the system, and ensures that they continue to meet the ever changing needs of their student population.

Adena Miller, M.A.
I'm not sure that school size or enrollment have an impact on whether RtI can be implemented effectively. However, in my experience, it does effect HOW RtI is implemented. For example, in small/rural schools, you may have only one teacher per grade level, or even one teacher for every two grade levels. In this system, the Universal problem-solving team will likely be made up of the same people as the individual problem-solving team. The conversations might be a little different (for example, horizontal alignment of universal instruction becomes moot), but both conversations are still critical.

Adena Miller, M.A.
This question really depends on the site, and particularly guidance from the district and the state. Ideally, educational specialists (including special educators), are involved in all processes, where appropriate.

Adena Miller, M.A.
I think you can have initial conversatons with a few people for planning purposes, but RtI is really about systemic reform. Therefore, you will need to take the time to understand your existing system, where gaps occur, and plan around those. If you do not have consensus with the majority of your staff, the efforts are not likely to gain traction. This is why the emerging stage is the first stage on our rubrics. You may consider using existing time built into your schedule to have these initial conversations. Staff meetings or department meetings could be a good place to start the exploration.