CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Student Assessment and Placement Systems Using Multiple Measures Elisabeth Barnett, CCRC NYS Student Success Center November 2018
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Agenda • Why use multiple measures for placement • Selection of a multiple measures system • Results of the SUNY research • Q and A
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Students needing 1+ developmental education course (NCES, 2013) 100% 80% 68% 60% 40% 40% 20% 0% Community Colleges Open Access 4-Year Colleges 3
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Community college 8-year graduation rates (Attewell, Lavin, Domina, and Levey, 2006) 50% 43% 40% 28% 30% 20% 10% 0% Students Needing RemediaEon Students Not Needing RemediaEon 4
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Under-placement and Over-placement Placement According to Exam Developmental College Level Student Ability ü Over-placed Developmental (English – 5%) (Math – 6%) ü Under-placed College Level (English – 29%) (Math – 18%) 5
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS COLLEGE 2: ENGLISH COLLEGE 2: MATH 0.2 0.2 0.18 0.18 0.16 0.16 14.5% 0.14 0.14 12.0% 0.12 0.12 9.9% 0.1 0.1 7.5% 0.08 0.08 4.8% 0.06 0.06 3.8% 0.04 0.04 2.7% 1.0% 0.02 0.02 0 0 GPA only Test only GPA and test Full model GPA only Test only GPA and test Full model 6
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Model R-Squared Statistics English R-Squared StaEsEcs – Graphical RepresentaEon 0.12 0.1 0.08 0.06 0.04 0.02 0 College 1 College 2 College 3 College 4 College 5 College 6 College 7 GPA ACCUPLACER GPA + ACCUPLACER Full Model Slides available at: bit.ly/capr_ashe16 7
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Model R-Squared Statistics Math R-Squared StaEsEcs – Graphical RepresentaEon 0.25 0.2 0.15 0.1 0.05 0 College 1 College 2 College3 College 4 College 5 College 6 College 7 GPA ACCUPLACER GPA + ACCUPLACER Full Model Slides available at: bit.ly/capr_ashe16 8
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Conclusions so far • Students placed into developmental education are less likely to complete. • Better assessment systems are needed. • HS GPA is the best predictor of success in college math and English. 9
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Multiple Measures Assessment 10
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Why Use Multiple Measures • Existing placement tests are not good predictors of success in college courses. • More information improves most predictions. • Different measures may be needed to best place specific student groups. 11
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Multiple Measures Options MEASURES SYSTEMS OR APPROACHES PLACEMENTS Administered by college: Waiver system Placement into • • 1. TradiEonal or alternaEve Decision bands tradiEonal courses • placement tests Placement formula Placement into • • 2. Non-cogniEve assessments (algorithm) alternaEve 3. Computer skills or career Decision rules coursework • inventory Directed self-placement Placement into • • 4. WriEng assessments support services 5. QuesEonnaire items Obtained from elsewhere: 1. High school GPA 2. Other HS transcript informaEon (courses taken, course grades) 3. Standardized test results (e.g., ACT, SAT, Smarter Balanced) 12
Possible Measures Type Examples Placement test Accuplacer ALEKS High school GPA, course grades, Self-report test scores From transcript Non-cogniEve assessments GRIT QuesEonnaire SuccessNavigator or Engage Career inventory, computer skills Kuder Career Assessment Home grown computer skills test WriEng examples Faculty-assessed porgolio Home-grown wriEng assessment 13
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Sources of HS transcript data Self-report research UC admissions uses self-report but • • The students bring a verifies aier admission. In 2008, at 9 transcript. campuses, 60,000 students. No campus had >5 discrepancies b/w • The high school sends. reported grades and student transcripts (Heks, 2016) • Obtained from state data files. College Board: Shawn & Maken, • • Self report. 2009: “Students are quite accurate in reporEng their HSGPA”, r = .73. ACT research oien uses self-reported • GPA and generally find it to highly Note: Consider using the 11 th correlate with students actual GPA: grade GPA. ACT, 2013: r = .84. 14
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Non-cognitive assessments Development of non-cognitive skills promotes students’ ability to think cogently about information, manage their time, get along with peers and instructors, persist through difficulties, and navigate the landscape of college…(Conley, 2010). Non-cognitive assessments may be of particular value for: Nontraditional (older) students. • Students without a high school record. • Students close to the cut-off on a test. • 15
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS NC 1: Success Navigator NC 2: Engage Domains: Domains: Academic discipline, commitment, • • MoEvaEon and skills, social self-management, support, social engagement, self-regulaEon supports Academic Success Index , includes: Advisor report also has: Projected 1 st year GPA • • Academic Success Index Probability of returning next • • RetenEon Index semester Also, Course Accelera3on Indicator CorrelaEon with GPA and retenEon, especially MoEvaEon scale. RecommendaEon for math or English • acceleraEon 16
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS NC 4: Learning and Study NC 3: Grit Scale Strategies Inventory (LASSI) Domains: Domains Grit and self-control. • Anxiety, aqtude, concentraEon, • informaEon processing, moEvaEon, selecEng main ideas, Provides score 1-5 on level of grit, self-tesEng, test strategies, Eme with 5 as maximum (extremely management, using academic griky) and 1 as lowest (not all griky). resources. CorrelaEon with GPA and conscienEousness CorrelaEon with GPA and retenEon. 17
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Concerns about the HS GPA (with thanks to John Hetts, 2016) Our test is different/better/more awesome. • Students really need developmental education. • High school GPA is only predictive for recent graduates. • Different high schools grade differently. • 18
Our test is different/better/more CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS awesome. NC ENGLISH NC MATH From BosEan (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012)
Students would be better off going CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS through developmental education. Developmental education student outcomes (Results from 8 studies, CCRC analysis 2015) 32 35 30 25 19 PosiEve 20 15 Null 15 10 NegaEve 6 5 2 5 0 Higher Level Students Lower Level Students 20
HS GPA is a beEer predictor than test results for long Hme (from HeEs, 2016) MMAP (in preparaEon): correlaEons b/w predictor and success (C or beker) in transfer-level course by # of semesters since HS
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS For the most part, college grades stay parallel with feeder high school grades. (Bostian, 2016)
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Ways to Combine Measures Algorithms: • – Placement determined by predictive model Decision Rules: • – New exemptions, cutoffs Decision Bands: • – “Bumping up” those in a test score range Directed Self-placement: • – Provide students with information; let them decide where they fit.
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Algorithm Example College Level Placement High Yes HS Record, ResulEng Accuplacer, Non- No Probability ExempEons? Cog data fed into of Success Algorithm Student Low Applies Remedial Level Placement
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Decision-Rule Example College Level Placement High High Yes HS Record and/ Low Accuplacer No or Non-Cog ExempEons? Test Performance? Student Low Applies Remedial Level Placement
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Decision-Band Example College Level Placement Above Band High Yes HS Record and/ Decision Accuplacer No or Non-Cog ExempEons? Band Test Performance? Student Band Below Low Applies Remedial Level Placement
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS The CAPR Assessment Study 27
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Organization of CAPR MDR DRC CCRC CCRC Descriptive Study of Evaluation of The New Evaluation of New Developmental Mathways Project Assessment Practices Education (RCT in TX) (RCT in NY) Supplemental Studies
CAPR CENTER FOR THE ANALYSIS OF POSTSECONDARY READINESS Research on Alternative Placement Systems (RAPS) • 5 year project; 7 SUNY community colleges • Evaluation of the use of predictive analytics in student placement decisions. • Random assignment/implementation/cost study • Current status: about to do final analysis 29
Recommend
More recommend
