GRADUATE NYC! Academy for Leaders in the Field of College - PDF document

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition GRADUATE NYC! Academy for Leaders in the Field of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 2 COLLECTING PARTICIPANT OPINION DATA: SURVEYS, INTERVIEWS, AND FOCUS GROUPS PUBLICLY AVAILABLE DATA SETS Stefanie Bruno Coordinator of Research & Evaluation CUNY Collaborative Programs Adenike Huggins Policy & Research Analyst CUNY School Support/Research & Evaluation May 24, 2012 3 Graduate NYC! Academy for Leaders in the Field of College Transition What Is The Best Way to Collect the Information? Method Purpose Advantage Challenges Surveys Quickly and/or easily Easy to compare and Might not get careful get lots of information analyze; can get lots in-depth feedback; from people in a non- of data; many sample Need to pretest threatening way questions already questions; requires exist; standardized ability to read; might be hard to deliver Face-to-Face Fully understand an Get full range and Time-consuming; hard Interviews individual’s opinions or depth of information; to analyze and experiences, or learn can be flexible compare data; can be more about their costly; interviewer can answers to surveys bias responses Focus Groups Explore a topic in Efficient way to get a Need skilled depth through group range and depth of moderator; have to discussion information in a short listen to a lot of time; new perspective different/overlapping may emerge voices; responses bias; expensive 1

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 4 What Is a Survey? • A method of gathering information from a number of individuals, a “sample”, in order to learn something about the larger population from which the sample has been drawn. • Provide a speedy and cost effective means of learning about people’s knowledge, attitudes, beliefs, and behavior. Graduate NYC! Academy for Leaders in the Field of College Transition 5 Pros and Cons of Conducting Surveys Surveys: Advantages • Easy to administer to a wide sample • Could assess changes in attitudes and behavior over time • Provide an efficient summary of opinions (if the population is sampled appropriately) • Highly structured, which reduces bias • Easy to analyze Surveys: Disadvantages • Yields mostly surface-level reactions • Difficult to ensure a representative sample respond 6 Graduate NYC! Academy for Leaders in the Field of College Transition Data That Can and Cannot be Obtained From Surveys Data that CAN be obtained: • Demographics • Desires • Expectations • Opinions • Perceived knowledge • Practices • Reactions Data that CANNOT be obtained: • Actual behavior and knowledge 2

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 7 Things to Consider Before Conducting a Survey • What is the purpose of the survey, what are you interested in learning? • Who is the best source for the data? • What is the best way to reach your target participants? • Who will be administering the survey? • Who will be affected by the results of your survey? • Who is the audience? • How will the results be used? Graduate NYC! Academy for Leaders in the Field of College Transition 8 Survey Creation • Include an introduction or cover letter and instructions • Start with an interesting/important question • Logical flow of questions: • Group similar questions • Smooth transitions • Ideal length • Leave white space • Demographics • Closing 9 Graduate NYC! Academy for Leaders in the Field of College Transition Survey Creation • Question Writing Tips • Include relevant decision-makers in the question design process • Keep the survey and individual questions short • Make sure that all of the questions are relevant • Use simple and direct language • Be neutral • Ask for one piece of information at a time • Balance rating scales • Put the response options in a logical order • If possible, pre-test the survey 3

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 10 What Is Wrong With This Question? I got along well with the teacher and students in my class. 1. Have you heard back yet from your top choice college? 2. I have a strong understanding of the physical sciences. 3. 4. Do you think the AHC math class is better than the other high school math classes you have taken? What is your current age? 5. 15-18 a) 18-20 b) 20-22 c) Graduate NYC! Academy for Leaders in the Field of College Transition 11 Target Sample Size • How many survey respondents do I need? • Survey purpose • Desired confidence level • Margin of error (confidence interval) • Total population size • http://www.custominsight.com/articles/random-sample- calculator.asp 12 Graduate NYC! Academy for Leaders in the Field of College Transition Target Sample Size Example Survey Question: Please check whether you agree or disagree with the following statement: “I am aware of various college entrance requirements.” __ Agree __ Disagree Let’s say 75% of those asked said “agree.” Analysis at confidence interval 5: We can say with 95% certainty that between 70 and 80 % (75 plus/minus 5) of the population in the school agree that they are aware of various college entrance requirements. Analysis at confidence interval 2: We can say with 95% certainty that between 73 and 77 % (75 plus/minus 2) of the school agree that they are aware of various college entrance requirements. 4

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 13 Maximizing Response Rate • Build a relationship with respondents • Include a personalized cover letter • Provide survey instructions • Interesting/relevant survey topic • Logical survey design • Send reminders • Offer a gift or incentive • Offer to share a summary of the results • Include your contact information • Use a professional reply email address • Indicate how long the survey takes to complete and a cutoff date • Timing Graduate NYC! Academy for Leaders in the Field of College Transition 14 Types of Surveys and Administration Methods Type Administration Tools Paper-and-pencil Mail, face-to-face, MS Word, Remark classroom, doorstep OMR Electronic Email, website link, Survey Monkey, Poll classroom Everywhere Verbal Telephone, face-to- MS Word, CATI face (Computer-Assisted Telephone Interviewing) Software • Who administers the survey? • How could this impact the results? • How much staff will be needed? 15 Graduate NYC! Academy for Leaders in the Field of College Transition Next Workshop: Survey Analysis • E.g. of what will be covered % of Students Who Claim to Know About How to Get Financial Aid for College 100% 85% 80% 58% 56% 60% 52% 43% 2011 40% 34% 2012 20% 0% n=504 n=607 n=1099 n=908 n=607 n=556 Current 10th Grade Current 12th Grade Current 11th Grade Cohort Cohort Cohort 5

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 16 Interviews Activity: Pair up with someone else in the room and interview each other. • Interview topic: college-readiness • Time allotted for activity: 10 minutes Graduate NYC! Academy for Leaders in the Field of College Transition 17 Interviews Tips • Start with a non-threatening question to build rapport • Stay silent to give the respondent time to think • Try to avoid questions that lead to one word responses • Break down a complex question into multiple questions • Ask questions about the intensity of a feeling • Ensure you strike a balance of positive/negative questions • Actively listen to the respondent 18 Graduate NYC! Academy for Leaders in the Field of College Transition Focus Groups Selecting Participants • Groups should consist of people with similar experiences so that they feel comfortable talking with one another • Strive for 6-10 participants per group • It is ok to select based on the purpose of the groups rather than strive for a representative sample 6

Graduate NYC! Academy for Leaders in the Field 5/23/2012 of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition 19 Focus Groups Deciding how many participants are needed: • Budget • Variety of relevant characteristics • Conduct as many groups as necessary until no new information is revealed Graduate NYC! Academy for Leaders in the Field of College Transition 20 Focus Groups Conducting the session • Schedule 1-1 1/2 hours • Try to map out how much time you want to spend on each question • Write 8-12 main questions with probes • Use a skilled moderator and assistant • Select a comfortable setting that is easily accessible • Audio record (after obtaining the proper permission) 7