Multiple Measures Assessment of Student Readiness for College - PowerPoint PPT Presentation

Multiple Measures Assessment of Student Readiness for College Courses Elisabeth Barnett, CCRC and NCREST Jennifer Kim, NCREST NACEP October 2017

Agenda • The college-readiness assessment landscape and the emerging use of multiple measures assessment • Middle and early colleges’ current assessment practices • Discussion and Q&A

Organization of CAPR CCRC CCRC MDRC Descriptive Study of Evaluation of The New Evaluation of New Developmental Mathways Project Assessment Practices Education (RCT in TX) (RCT in NY) Supplemental Studies

Research on an Alternative Placement Strategy using Multiple Measures • 5 year study; seven colleges; 10,000 students • Alternative placement system • Random assignment: half of students placed using algorithm • Analysis of student outcomes and implementation. 4

RAPS – Partner Sites A – CAPR/CCRC/MDRC B – Cayuga CC C – Jefferson CC D – Niagara County CC E – Onondaga CC F – Rockland CC G – Schenectady County CC H – Westchester CC Slides available at: bit.ly/capr_ashe16 5

Predictive analytics used to develop an algorithm Use data from Develop formula to previous predict student cohorts performance Use formula to place entering cohort of students 6

Why Use Multiple Measures Assessment? 7

Community college 8-year graduation rates (Attewell, Lavin, Domina, and Levey, 2006) 50% 43% 40% 28% 30% 20% 10% 0% Students Needing Remediation Students Not Needing Remediation 8

Under-placement and Over-placement (severe) Placement According to Exam Developmental College Level  Student Ability Over-placed Developmental (English – 5%) (Math – 6%)  Under-placed College Level (English – 29%) (Math – 18%) 9

COLLEGE 2: ENGLISH COLLEGE 2: MATH 0.2 0.2 0.18 0.18 0.16 0.16 14.5% 0.14 0.14 12.0% 0.12 0.12 9.9% 0.1 0.1 7.5% 0.08 0.08 0.06 0.06 4.8% 3.8% 0.04 0.04 2.7% 1.0% 0.02 0.02 0 0 GPA only Test only GPA and test Full model GPA only Test only GPA and test Full model 10

Model R-Squared Statistics English R-Squared Statistics – Graphical Representation 0.12 0.1 0.08 0.06 0.04 0.02 0 College 1 College 2 College 3 College 4 College 5 College 6 College 7 GPA ACCUPLACER GPA + ACCUPLACER Full Model Slides available at: bit.ly/capr_ashe16 11

Model R-Squared Statistics Math R-Squared Statistics – Graphical Representation 0.25 0.2 0.15 0.1 0.05 0 College 1 College 2 College3 College 4 College 5 College 6 College 7 GPA ACCUPLACER GPA + ACCUPLACER Full Model Slides available at: bit.ly/capr_ashe16 12

Conclusions so far • Better assessment systems are needed. • HS GPA is the best predictor of success in college math and English. 13

Multiple Measures Assessment 14

Why Use Multiple Measures • Existing placement tests are not good predictors of success in college courses. • More information improves most predictions. • Different measures may be needed to best place specific student groups. 15

Current assessment practices in the US (preliminary; CAPR 2017) % using….. Math English Standardized tests 88% 87% High school performance 40% 37% Planned course of study 29% 18% Other indicators of motivation 11% 14% No assessment done 6% 6% 16

Possible Measures Type Examples Placement test Accuplacer ALEKS High school GPA, course grades, From transcript test scores Self-report From SAT, SAT, SB, etc. Non-cognitive assessments GRIT Questionnaire SuccessNavigator or Engage Career inventory, computer skills Kuder Career Assessment Home grown computer skills test Writing examples Faculty-assessed portfolio Home-grown writing assessment Individual advisement 17

Non-cognitive assessments Development of non-cognitive skills promotes students’ ability to think cogently about information, manage their time, get along with peers and instructors, persist through difficulties, and navigate the landscape of college…(Conley, 2010). Non-cognitive assessments may be of particular value for: • Nontraditional (older) students. • Students without a high school record. • Students close to the cut-off on a test. 18

Multiple Measures Options (Barnett and Reddy, 2017) MEASURES SYSTEMS OR APPROACHES PLACEMENTS • • Administered by college: Waiver system Placement into • 1. Traditional or alternative Decision bands traditional courses • • placement tests Placement formula Placement into 2. Non-cognitive assessments (algorithm) alternative • 3. Computer skills or career Decision rules coursework • • inventory Directed self-placement Placement into 4. Writing assessments support services 5. Questionnaire items Obtained from elsewhere: 1. High school GPA 2. Other HS transcript information (courses taken, course grades) 3. Standardized test results (e.g., ACT, SAT, Smarter Balanced) 19

But what about …. • Students placed via multiple measures will not be successful. • Our test is different/better/more awesome. • High school GPA is only predictive for recent graduates. • High schools grade differently. • It’s too hard to get or use transcripts/it’s not worth it. 20

Students placed via multiple measures will likely be successful. Davidson County, NC, 2013-15 Ivy Tech, IN, 2014-15 100% 100% 76% 80% 80% 68% 65% 64% 64% 59% 59% 57% 57% 60% 60% 48% 40% 40% 20% 20% 0% 0% English Math Reading English Math Comparison HS Data Accuplacer HS Data Rule s use d fo r E ng lish and Math: HSGPA >=2.6 and Rule s use d fo r E ng lish and Math: HSGPA >=2.6 c o mple tio n o f fo ur ye ars o f mathe matic s inc luding o ne ye ar b e yo nd Alg e b ra 2 in HS F ROM HE T T S, 2016 21

Our test is different/better/more awesome. NC ENGLISH NC MATH From Bostian (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012 – see also Table 1)

HS GPA is a better predictor than test results for long time (Hetts, 2016) MMAP (in preparation): correlations b/w predictor and success (C or better) in transfer-level course by # of semesters since HS

For the most part, college grades stay parallel with feeder high school grades. (Bostian, 2016)

Sources of HS transcript data Self-report research • UC admissions uses self-report but • The students bring a verifies after admission. In 2008, at 9 transcript. campuses, 60,000 students. No campus had >5 discrepancies b/w The high school sends. • reported grades and student transcripts (Hetts, 2016) Obtained from state data files. • • College Board: Shawn & Matten, • Self report. 2009: “Students are quite accurate in reporting their HSGPA”, r = .73. • ACT research often uses self-reported GPA and generally find it to highly Note: Consider using the 11 th correlated with students actual GPA: grade GPA. ACT, 2013: r = .84. 25

Examples of alternative assessment systems • Indiana • California • North Carolina • New York • Minnesota • Wisconsin 26

IVY TECH, INDIANA – WAIVERS (CCCSE, 2016) I vy Tech Community College (I N) has been using a multiple measures placement policy for degree-seeking students since 2003. • Students may submit any of the following documents: ACT, SAT, or PSAT scores for tests taken within the past four years • The college added the acceptance of high school GPA to the policy for students entering the college in fall 2014. • Students who do not have the above documentation are required to take the college’s custom ACCUPLACER diagnostic assessment. 27

IVY TECH PASS RATES 28

CALIFORNIA MMAP Project (Hetts, 2016) • Collaborative effort of CCCCO, Common Assessment Initiative (CAI), Cal- PASS Plus, RP Group and ~ 45 CCC pilot colleges* • Identify, analyze, & validate multiple measures data • For English, Mathematics, ESL and Reading • Focus on predictive validity (success in course) • Key variables included HSGPA, last course in discipline, course grade or level, AP course-taking, CST scores, etc. bit.ly/MMAP2015 and http://bit.ly/MMAPRules 29

Example– CA Math Placement Level Direct Matriculants (from HS) Non-Direct Matriculants Calculus I 11th- grade GPA ≥ 3.6 12th- grade GPA ≥ 3.1 and took Calculus Passed Precalculus or 11th- grade GPA ≥ 3.2 and Trigonometry (or better) Precalculus C (or better) 12th- grade GPA ≥ 3.5 12th- grade GPA ≥ 3.3 11th- grade GPA ≥ 3.4 12th- grade GPA ≥ 3 and Algebra II Precalculus California Standards Test ≥ 340 11th- grade GPA ≥ 2.6 and took Passed Algebra II (or better) Calculus 12th- grade GPA ≥ 3 and Calculus C (or better) 11th- grade GPA ≥ 3.4 12th- grade GPA ≥ 3.3 11th- grade GPA ≥ 3 and Trigonometry Precalculus C+ (or better) 12th- grade GPA ≥ 2.8 and Passed Algebra II (or better) Precalculus C (or better) 11th- grade GPA ≥ 3 and Algebra II B (or better) 11th- grade GPA ≥ 3.2 12th- grade GPA ≥ 3.2 College Algebra 30 11th- grade GPA ≥ 2.9 and 12th- grade GPA ≥ 3.0 and Precalculus C (or better) ( ) (