GRADUATE NYC! Academy for Leaders in the Field of College - PDF document

5/4/2012 GRADUATE NYC! Academy for Leaders in the Field of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition Asking the Right Questions Taking a deep look at your data collection techniques and how the choices you make impact data quality and program implementation practices. Graduate NYC! Academy for Leaders in the Field of College Transition Revising Day 1: A Recap Program Management & Evaluation/Assessment Data Overview: What do we collect? Why do we collect it? Introduction to a Logic Model Opportunity for Feedback & Reflection 1

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition Day 2: What to Expect Asking the Right Questions: A Case Study Data Types & Collection Strategies In-depth Look at Program/Administrative Data Data Quality, Ethics, and Security Graduate NYC! Academy for Leaders in the Field of College Transition Academy Overview Day 1: Introduction to Data & Logic Models Day 2: Answering Questions with Administrative Data Day 3: The Role of Surveys, Focus Groups, Interviews, & Existing Publicly Available Data Day 4: Data Management & Analysis Day 5: Communicating through Data Graduate NYC! Academy for Leaders in the Field of College Transition A Case Study What data do you need to collect for this program and why? 2

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition What do we collect and how? • Data Types • Collection Methods • Demographic Data • Administrative Records • Participants’ background • Information/Registration Forms information • Surveys • Program Data • Interviews/Focus Groups • Begin and end dates of • Other Methods? program participation, level of services provided, attendance • Outcomes Data • College matriculation, post- participation employment, civic involvement Graduate NYC! Academy for Leaders in the Field of College Transition Deciding on a Method: Considerations • Staff resources • Time, expertise • Costs • Participation incentives, dedicated data collection/entry staff • Time constraints • Urgent data request • Frequency of collection • One-time data request • Ongoing evaluation • Purpose of data • Funder request • Internal research Graduate NYC! Academy for Leaders in the Field of College Transition A Framework for Data Collection & Use: • Utility • Do the data address the questions that are important to your program? • Is any of the information unnecessary or redundant? 3

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition A Framework for Data Collection & Use: • Timeliness • Have you discussed appropriate timelines for data collection and data entry? • Are data collection processes designed to give you data quickly enough to make program decisions? Graduate NYC! Academy for Leaders in the Field of College Transition A Framework for Data Collection & Use: • Quality • Are standards of quality established and communicated for all phases of research process? Graduate NYC! Academy for Leaders in the Field of College Transition A Framework for Data Collection & Use: • Security • Is student confidentiality enforced? • When appropriate, are students de-identified? • Have you obtained consent and assent? 4

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition A Framework for Data Collection: Utility Quality Timeliness Security Graduate NYC! Academy for Leaders in the Field of College Transition Dilemmas of Practice Utility Timeliness Quality Security Graduate NYC! Academy for Leaders in the Field of College Transition Example Your funder requires that you submit evidence that your students are program eligible based on their enrollment in a civics class in high school. What process would you use to determine the best data collection method for this request? 5

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition Administrative and Program Data • What information? • Existing administrative data • Transcript data • Program information/registration forms • Other data collection • Staff records the number of advising sessions a student attends • Others • Why collect it? • What is the utility of the data? • How will the data be used? • Who will see the results? Graduate NYC! Academy for Leaders in the Field of College Transition Obtaining & Using Existing Administrative Data Advantages Challenges • Cost effective • Lack of control over data • Data are already available elements • Easy to store (if electronic) • Varying degrees of • Historical data completeness • Construction of comparison • Limited access group • Data agreements • Compelling to funders and • Large data files outside audiences • Confusing data codes • Tracking participants • Data Verification Graduate NYC! Academy for Leaders in the Field of College Transition Collecting Program Data from Students • How do you get information about your students? • Information/Registration forms (Example: MSP) • Utility • Is your “instrument” answering the questions that you need to know? • Timeliness • How much time lapses between collection and entry? • Are data collected at one time, or is collection spread out over a period of time? • Challenges • Student-reported data • Informal data collection • Mixed methods • Decentralized data collection 6

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition Data Storage • Excel or Flat file data storage • Organized around a single table • Easy to sort data and do analyses using functions • Especially useful for numeric data • Good if interested in only one type of data (eg. academic coursework) • Difficult to manage large datasets • Data duplication – Can have implications on data quality • Access or Relational database • Organized around the relationships between tables • Keeping track of several different types of data (eg. demographic, academic activities, non-academic activities) • Easy to run complex queries; which combine the data from the various tables in the database • Other database options • Online options Graduate NYC! Academy for Leaders in the Field of College Transition Data Storage Tips • Employ uniform field layouts and formats • Use consistent naming conventions • Format consistently (dates, numbers, and text fields) • Insert drop-down menus to minimize data entry error • Restrict maximum number of characters • Employ accountability measures • Include fields indicating when and by whom records are added or updated • Prevent data redundancy • If something appears in one table, it doesn’t need to appear in another table • Prevent duplicate records • Create primary keys • Back up your data Graduate NYC! Academy for Leaders in the Field of College Transition Ensuring Data Quality • Establishing a culture of quality • Beliefs • Good data are an integral part of teaching and learning • Everyone involved in a program is responsible for quality data • It is possible to create orderly information from disorderly settings • Components • Policies and Regulations • Standards and Guidelines • Training and Professional Development • Timelines and Calendars • Technology • Data Entry Environment National Forum on Education Statistics 7

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition Factors Components of Affecting a a Quality Data Quality Data Culture Culture Quality Data Culture Roles in a Importance of a Quality Data Quality Data Culture Culture Graduate NYC! Academy for Leaders in the Field of College Transition Data Quality Strategies • Getting buy-in: How to get everyone excited about data?! • Everyone should get a sense of his/her role in the big picture • Feedback loop • Create an open dialogue about the processes • Encourage data entry staff to openly share what works and what doesn’t • Share the exciting results of the research with data entry staff! • Data checklists • Are the data complete? • Can you obtain missing data? • Are data reasonable (no zeros where zero is an impossible response)? Graduate NYC! Academy for Leaders in the Field of College Transition Feedback Loop Data Entry Staff Share Identify findings concerns with during data data entry entry staff process Administrators/ Supervisors 8

5/4/2012 Graduate NYC! Academy for Leaders in the Field of College Transition Bad Data: Why Does it Matter? Percentage of Male Students in College Program 50 45 40 35 30 25 20 2007 2008 2009 2010 Graduate NYC! Academy for Leaders in the Field of College Transition Bad Data: Why Does it Matter? Percentage of Male Students in College Program 100 90 80 70 60 50 40 30 20 10 0 2007 2008 2009 2010 Graduate NYC! Academy for Leaders in the Field of College Transition Bad Data: Why Does it Matter? Percentage of Male Students in College Program 100 Female 90 Male 80 70 60 50 40 30 20 10 0 2007 2008 2009 2010 9