Understanding the First-Year Experience Four Ways to Generate Data- - PowerPoint PPT Presentation

Understanding the First-Year Experience Four Ways to Generate Data- Driven, Actionable Insights Steve Wygant, Danny Olsen, Bryce Bunting Brigham Young University 26 th International Conference on the First-Year Experience Waikoloa, Hawaii, June 2013

Introduction : Brigham Young University  Private, faith-based institution in Provo Utah, USA  30,000+ undergraduates  Academically competitive:  Mean HS GPA (2012): 3.81 (high=4.0)  Mean ACT/SAT (2012): 28/1270 2

Academics  Strong programs in accounting, business, engineering, animation, music, etc.  Ranked #9 in Baccalaureate-origins of U.S. Doctorate Recipients  Ranked #68 in US News & World Report 2013 Best Colleges 3

International focus  60 languages taught  75% of students have a second language  33% enroll in a foreign language course each semester  7% go on Study Abroad  6% are International Students  Outreach efforts around the world 4

2012 First Year Class  2,455 Women  2,975 Men  5,430 Total  55% of applicants are admitted  80% of admitted enroll 5

Retention 89% retained (full-time first-time, Fall 2011- Fall 2012) Graduation 4 year - 31% 6 year - 78% 8 year - 86% (2004 cohort as of 2012) 6

Athletics 7

A Tale of Two Students 8

Guiding questions  Why do intelligent, capable students flounder?  Academic warning/probation  Fail to achieve goals, live up to potential  Can we identify students who will flounder?  Using data obtained before students arrive on campus?  Using data obtained during the course of the school year?  How could and should such data be used?  Individually, and as a campus community, how do we help students flourish?  Can data help us identify student motivations, strengths and challenges in order to connect them with resources they need?  How would such data be used? 9

As we get going …  Focus on processes, not specific findings  Questions you might ask yourselves  “How would processes like this work with our campus and our students?”  “What data do we have that could help us predict student success?”  “What other data can we gather that might help us predict student success?”  Session as conversation  Ask questions  Share your practice 10

Survey instruments used  Pre-arrival Survey  High school experiences & practices  Expectations for college  Self-descriptions  End-of-year Survey  Engagement: work ethic, participation  Attitudes, practices  Self-descriptions  Outcomes, influences 11

Getting and using good data 12

Four ways of looking at data 1. Before-after • How do attitudes & behaviors change? Expectations vs. experience? 2. Flourish-flounder • How are students who are thriving by the end of the first year different from those who are not? 3. Over-achieving vs. underachieving • How are students who exceed academic expectations different from those who perform below expectations? 4. Cluster analysis • How do sub-groups of students who begin the first year with different attitudes and motivations differ in behavior and performance? 13

Before-after analysis Expected vs. actual hours spent per weekday ( Expected hours from Pre-arrival survey [PA], Actual hours from End-of-year survey [EOY]) PA EOY Using internet/phone (not school work) Extra-curricular activities Working for pay Academic work outside of class Attending classes Very low Low Moderate High Very high 14

Before-after analysis Percent of students who rated themselves as "Very competent" or "Quite competent" PA EOY Pre EOY Post think about problems in evaluate the credibility of explore a variety of ways evaluate your own original ways information sources to solve a problem assumptions, conclusions and reasoning 15

Flourish-flounder analysis Flourish - flounder (End-of-year survey) Flourishing Floundering Getting Agree (low) - Disagree (high) by strongly somewhat somewhat badly (Pre-arrival survey) … satisfied with my life … like the kind of person I am … able to make friends easily …. know & understand myself Very low Low Moderate High Very high 16

Flourish-flounder analysis Flourishing-floundering (End of school year) Flourishing Getting Floundering Agree (low) - Disagree (high) by strongly somewhat somewhat badly (on Pre-arrival survey) BYU was the right choice I will be able to make friends at BYU I will do well academically at BYU Very low Very high Low Moderate High 17

Flourish-flounder analysis Flourishing-floundering (End of school year) Floundering Flourishing Getting Frequency during first year strongly somewhat by somewhat badly Adapt study habits & strategies Persist in doing class work Procrastinate doing class work Very low Low Moderate High Very high 18

Underachieving vs overachieving Comparing students whose first-year GPA was higher than expected to those whose GPA was lower than expected. (Expected GPA [EGPA] is based on ACT/SAT and HSGPA.)  Badly underachieve : GPA 1/2 point or more lower than EGPA (Example: EGPA=3.75, GPA<=3.25)  Somewhat underachieve : GPA ¼ to ½ point below EGPA  Matched : GPA within ¼ grade of EGPA  Somewhat overachieve : GPA ¼ to ½ point higher than EGPA  Strongly overachieve : GPA more than ½ point above EGPA 19

Underachieving vs overachieving Underachieving (GPA < expected) vs overachieving (GPA > expected) Overachieved Underachieved Matched High school experience badly somewhat somewhat strongly Flourished (1) or floundered (5) Hours/week studied last year Did readings, assignments before class Academic success: Ability (1) v. effort (5) Procrastinated doing class work Moderate Very low Low High Very high 20

Underachieving vs overachieving Underachieving (GPA < expected) vs overachieving (GPA > expected) Underachieved Overachieved Hours per typical weekday somewhat Matched in first year badly somewhat strongly academic work outside of class socializing in person on-line (not school work)/gaming working for pay Very low Low Moderate High Very high 21

Underachieving vs overachieving Underachieving (GPA < expected) vs overachieving (GPA > expected) Underachieved Overachieved Matched badly somewhat somewhat strongly Complete readings & assignments pre-class Get feedback before submitting a paper Work harder than you thought you could Seek help when you did not understand Adapt your study habits and strategies Persist in doing the work required Procrastinate doing the work needed Very low Low Moderate High Very high 22

Cluster analysis  Questions:  How do sub-groups among first-year students differ in ability, attitude, motivation?  How do those differences affect behavior and performance?  Identified subgroups using cluster analysis  Process  Factor analysis of Pre-arrival survey questions  Create scales using multiple items from factor analysis  Use scales to look for clusters  Art and science 23

Sample items that form scales High school academic engagement (How frequently did you …) Adapt your study habits and strategies to help you do better in your classes Persist in doing the work required to succeed in your classes, even when it was difficult Seek help when you did not understand important ideas or concepts Spiritual/social reasons for choosing BYU (How important was … in your decision to choose BYU?) Getting a college education in a spiritually enriching environment Opportunity to associate with peers who have values and beliefs similar to yours Opportunity to develop a greater capacity to do good for others Social/academic engagement (How important will ... be during your first year at BYU?) Talking with faculty members outside of class Participating in study or discussion groups Discussing ideas from your reading or classes with others outside class 24

Sample scales that distinguish between clusters Cluster Scale AF AU Dis Bal High school academic engagement Very high Low Low High Spiritual/social reasons for choosing BYU Moderate Moderate Low High Importance of social/ academic engagement in FY Moderate Moderate Low High AF =Academically focused, AU =Academically undeveloped, Dis =Disengaged, Bal =Balanced 25

Clusters defined Academically focused (AF) 1. Very high on academic experience/motivation scale  Average on other scales  Academically undeveloped (AU) 2. Low on academic experience/motivation scale  Average on other scales  Disengaged/reticent (Dis) 3. Low on every scale  Balanced/well adjusted (Bal) 4. High on every scale  26

Cluster analysis Clusters Importance of college goals AF AU Dis Bal Socialize Learn new ideas and perspectives Enhance your personal spirituality Grow from being challenged intellectually Very low Low Moderate High Very high AF =Academically focused, AU =Academically undeveloped, Dis =Disengaged, Bal =Balanced 27

Cluster analysis Clusters First-year academic participation AF AU Dis Bal Complete readings and assignments Discuss course-related ideas outside class Get feedback before submitting work Work harder than you thought you could Write multiple drafts of a paper Very low Low Moderate High Very high AF =Academically focused, AU =Academically undeveloped, Dis =Disengaged, Bal =Balanced 28