Institutional research and decision support Marin Clarkberg Associate - PDF document

9/11/2019 Institutional research and decision support Marin Clarkberg Associate Vice Provost for Institutional Research & Planning Cornell University 1 Institutional Research • Research about the institution itself • Institutional researchers collect, analyze, report, and store data about their institution’s students, faculty, staff, curriculum, course offerings, and learning outcomes • Using data effectively to help make better decisions in universities • Supports excellence in delivering the mission 2 1

9/11/2019 “Institutional Effectiveness” and “Quality Assurance” • Nearly all U.S. universities have an office of institutional research • Some other countries have IE or QA and not IR 3 Institutional Research at Cornell The mission of Institutional Research & Planning (IRP) is to provide official, accurate, and unbiased information and analysis about the university in support of institutional planning, decision ‐ making, and reporting obligations. 4 2

9/11/2019 Institutional Research at Cornell The mission of Institutional Research & Planning (IRP) is to provide official, accurate, and unbiased information and analysis about the university in support of institutional planning, decision ‐ making, and reporting obligations. IRP supports “data ‐ informed decision making.” 5 Support excellence in delivering the mission “Cornell’s mission is to discover, preserve and disseminate knowledge, to educate the next generation of global citizens, and to promote a culture of broad inquiry throughout and beyond the Cornell community.” Learning . Discovery . Engagement . 6 3

9/11/2019 Learning. What leads to students’ academic success? 7 Learning. What leads to students’ academic difficulties ? • Lack of academic preparation • Lack of maturity, e.g. time management • Biting off more than one can chew • Too many extra curriculars • Taking too many classes 8 4

9/11/2019 The Data: What predicts academic success? • Tens of thousands of student records • Outcomes (that is, grades) in thousands of courses • We know a lot about the students (including things from their application to Cornell) • We know some about other things a student is doing • Where they live • Some extracurriculars • Other classes that they are taking Can we use what we know build data ‐ driven advising? 9 Advice to undergraduates Give yourself the opportunity to do well: • Don’t exceed 18 (or maybe 21?) credits in a semester • Don’t take all the hardest courses all at once That’s old ‐ fashioned human advice… what do the data say? 10 5

9/11/2019 A My theory B C D 12 18 11 This line says that every increase in A course load of four My theory credit hours is associated with an B increase in GPA of a full letter grade! C D 12 18 12 6

9/11/2019 A B C D 12 25 18 13 Spurious relationships Observed in the data Reason for the relationship Ice cream sales and death by drowning Both increase in hot summers Shoe size and reading ability in children Older children have bigger feet and read better than younger children Radiation therapy is associated with Cancer causes both the need for death radiation and death Course overloads are associated with Only the best students attempt to take a better grades course overload See also https://www.tylervigen.com/spurious ‐ correlations 14 7

9/11/2019 The problem with being “data ‐ driven” • Universities should not operate “like Amazon.” The student experience is complicated. Life advice is complicated. • It is difficult to isolate the effect of any given facet of the student experience (like course load… or a specific program at the university) • “Correlation is not causation” • Statistical relationships may overlook qualitative differences. For example, are credit hours taught really comparable constructs when they are taught in physics or history or art? 15 Okay, then why use data at all? • Anecdotal evidence is not enough. Life experiences, pet theories, and gut feelings can be biased. • Facts are an important part of building a shared understanding among multiple parties who don’t always agree. • Universities hone rhetorical skills, but decisions should not be based on force of argumentation, personal reputation or stature, or political connections. 16 8

9/11/2019 Data ‐ informed decisions take place in a cycle Commitment to unbiased, impartial inquiry Questions Assemble Insight data Analysis 17 Data ‐ informed decisions take place in a cycle Commitment to unbiased, impartial inquiry Questions Assemble Insight data Communication: Sharing analysis to Analysis support insight 18 9

9/11/2019 Enrollment from 1868 ‐ present Only to 1937 shown below… impossible to see this information in a meaningful way in a table 19 http://irp.dpb.cornell.edu/ 20 10

9/11/2019 Law School 21 Vet School 22 11

9/11/2019 How should data be displayed? Table Graph Data are expressed as text: words and Data are expressed graphically, as a numbers picture Best choice when the display will be Works best when the message you wish used to look up individual values or the to communicate resides in the shape of quantitative values must be precise. the data: patterns, trends, exceptions Accessible by screen readers Can be difficult for the visually impaired Sometimes, the choice is driven by the “client”… 23 Sometimes you want specific numbers… 24 12

9/11/2019 Maybe both Numbers and graphs can support each other 25 Wow? • Co ‐ Authorship on Pharmacology, Toxicology and Pharmaceutics Articles 26 13

9/11/2019 Is staff headcount increasing? 27 Headcount has increased… a bit. 28 14

9/11/2019 Unnecessary use of a third dimension 29 Misleading use of a third dimension 30 15

9/11/2019 This just in.. 31 Good analysis and visualization… • Clarifies patterns rather than distorts • Is designed with a purpose, to communicate particular findings • Facilitates making comparisons of various data elements • Change over time (such as in a line chart) • Differences across schools or colleges or programs (maybe stacked bar chart) • Helps to tell a story • Guides the viewer to think about patterns in data rather than thinking about graphic design frills or fancy software 32 16

9/11/2019 Data ‐ informed decisions take place in a cycle Commitment to unbiased, impartial inquiry Dialogue, Questions exploration Assemble Insight data Analysis 33 Importance of impartiality The mission of Institutional Research & Planning (IRP) is to provide official, accurate, and unbiased information and analysis about the university in support of institutional planning, decision ‐ making, and reporting obligations. • Facts are an important part of building a shared understanding among multiple parties who don’t always agree. • Universities hone rhetorical skills, but decisions should not be based on force of argumentation, personal reputation or stature, or political connections. 34 17

9/11/2019 Decisions are made by people… … but those decisions should be informed decisions • Data are not licensed to drive • Data do not speak for themselves • Data relationships need to be interrogated; “theory” is important • Data and analysis must be trusted and nonpartisan • Data and analysis must be effectively communicated—in a clear and nonpartisan manner—to be useful 35 Supporting decision ‐ making with IR • Skilled analysts and communicators with a deep understanding of higher education • A location in the institution that is separate from the functional offices that are generating the data and accountable for outcomes • A commitment to discovery with impartiality 36 18

9/11/2019 Thanks! Marin Clarkberg Associate Vice Provost for Institutional Research & Planning clarkberg@cornell.edu irp.dpb.cornell.edu 37 19