Session Three: Identify Measures of Success and Plan Cycles of Inquiry Lauren Bryant and Allison Black-Maier
In this session... 1. Revisit logic model from Session 2. Identify key outcomes and measures of success. 2. Check the practicality of your measures of success. 3. Establish a plan for conducting collaborative inquiry.
Revisiting our Logic Models Within your project team, consider the following: What are the key outcomes in your logic model? In other words, what outcomes will be the most important for you to track and measure? Why?
Practical Measures Today we’re going to focus on two kinds of practical measures: outcome measures and primary driver measures (Bryk et al., 2015).
Practical Measures Outcome measures operationalize your outcomes, precisely defining the common aim which allows project team members to have explicit conversations about the work of the project. This is the “end-of-the-line” indicator of the success of your project.
Examples of Outcome Measures For an RPP focusing on improving a developmental math program at the community college level: ● The percentage of students who enrolled in the program and passed both semesters with a C or better.
Examples of Outcome Measures For an RPP focusing on increasing eighth grade students’ participation in engineering: ● The percentage of students who took a year-long Introduction to Engineering course in eighth grade who go on to take engineering-related courses in high school. More long-term possibility: ● The percentage of students who took a year-long Introduction to Engineering course in eighth grade who go on to declare an engineering major in college.
Practical Measures Within your project team, consider the following: What are possible outcome measures of your key outcome(s)? What would operationalize your key outcome and serve as an end-of-the-line indicator of the success of your project?
Practical Measures Primary driver measures gather information on the major mechanisms that are going to result in your key outcomes. These measures should: 1. Predict your outcome measure. 2. Be sensitive enough to change to give you timely feedback. 3. Provide guidance on what to modify to improve. 4. Be folded into cycles of collaborative inquiry.
Primary Driver Measure Examples For an RPP focusing on improving a developmental math program at the community college level: ● A three-minute, 25 item questionnaire to measure students’ productive persistence.
Examples of Primary Driver Measures For an RPP focusing on increasing eighth grade students’ participation in engineering: ● A five-minute, 10-item questionnaire to measure students’ sense of belonging in engineering and interest in engineering.
Practical Measures What are possible primary driver measures for your project? What measures would meet the following guidelines? 1. Predict your outcome measure. 2. Be sensitive enough to change to give you timely feedback. 3. Provide guidance on what to modify to improve. 4. Be folded into cycles of collaborative inquiry.
Collaborative Cycles of Inquiry An important aspect of primary driver measures is that they won’t do you any good if they are not integrated into collaborative cycles of inquiry. However, collaborative cycles of inquiry do not evolve naturally in an RPP environment...
Collaborative Cycles of Inquiry Large project teams are not immune to the Second Law of Thermodynamics - that closed systems tend to move toward chaos. Reversing this tendency toward chaos requires the input of energy!
Collaborative Cycles of Inquiry Collaborative cycles of inquiry are basically smaller research cycles that you build into your project so that you can improve as you go. This approach is useful (arguably even necessary) for projects that are trying to tackle complex problems for which few to no workable solutions exist (Bryk et al., 2015; Patton, 2010).
Building Collaborative Cycles of Inquiry Team members should ask themselves the following questions: ● How often will you need to collect data using your outcome measure(s) ? ● How often will you need to collect data using your primary driver measure(s) ? ● Who should collect the data? Who will the data be collected from? ● Who should the results be shared with, and when? Includes people within your RPP and people external to your project. ● How and when will the results be used? What aspects of your project may change based on the results?
Examples of Collaborative Cycles of Inquiry For an RPP focusing on improving a developmental math program at the community college level:
Year One Rapid Cycle Annual Cycle Student grade data is collected one Students take 3-minute survey after week after the close of the semester every class , to be administered via email by Mark from the research team and push by Mark from the research team. analyzed by Jesse from the research Reminders for survey given by Professors team. Results are shared via email to Smith and Brown. the entire RPP team. Student data are analyzed every 3 RPP members will meet 3 weeks by Jesse from the research weeks after the close of team. Results are shared via email the semester to review the to all members of the RPP. student grade data and the final round of survey data. Team will discuss together Changes and what changes to make for Results are used immediately by modifications will be in the following year to better Professors Smith and Brown to plan place one week before the reach their key outcomes. subsequent lessons, and by campus start of Year Two. administrators in standing bi-weekly Professors Smith and Brown meeting to modify and strengthen will be responsible for external supports to students. changes to the curriculum, campus administrators will be responsible for changes to external supports.
Build Your Collaborative Cycles of Inquiry ● How often will you need to collect data using your outcome measure(s) ? ● How often will you need to collect data using your primary driver measure(s) ? ● Who should collect the data? Who will the data be collected from? ● Who should the results be shared with, and when? Includes people within your RPP and people external to your project. ● How and when will the results be used? What aspects of your project may change based on the results?
Gallery Walk
References Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How america’s schools can get better at getting better. Harvard Education Press: Cambridge, MA. Fishman, B. J., Penuel, W. R., Allen, A. R., & Cheng, B. H. (Eds.). (2013). Design-based implementation research: Theories, methods, and exemplars. Teachers College, Columbia University. Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.
Additional Resources on Cycles of Inquiry ● http://researchandpractice.org/toolkit-gather-and-use-evidence/ ● https://www.carnegiefoundation.org/blog/revisiting-purposes-practical-measur ement-improvement-learning-bten-measurement-system/ ● https://www.carnegiefoundation.org/blog/improvement-discipline-in-practice/ ● http://www.thecorecollaborative.com/impact-teams ● http://hepg.org/hel-home/issues/25_3/helarticle/improving-teaching-and-learni ng-through-instructio
Recommend
More recommend
