Evaluating an Alternative CS1 for Students with Prior Programming - PowerPoint PPT Presentation

Evaluating an Alternative CS1 for Students with Prior Programming Experience Michael S. Kirkpatrick Chris Mayfield SIGCSE Technical Symposium March 2017 Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

JMU Introductory Sequence Java Not Java CS1 CS2 CS 139 CS 239 CS 240 (4 cr.) (4 cr.) (3 cr.) Java • Development process • Classes/objects • New language • Control structures • OOP concepts • Recursion • Variables and expressions • Packages • Asymptotics • Functions • References • Searching • Arrays • Recursion • Sorting MA 205 • Classes/objects • Exceptions • Hashing Applied • Console I/O • Basic file I/O • Trees (3 cr.) • Design and testing Calculus • Collections Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

JMU Introductory Sequence CS 139 CS 239 CS 240 (4 cr.) (4 cr.) (3 cr.) CS1 CS1 CS2 CS2 Goals and benefits CS 139 CS 239 CS 240 • No prior exposure required (4 cr.) (4 cr.) (3 cr.) • Emphasis on algorithmic thinking • Maintain positive climate • Small class sizes • Exposure to two languages MA 205 MA 205 (3 cr.) (3 cr.) Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

JMU Introductory Sequence C students just Fall 2014: Prereq + Calculus Fall 2013: Accelerated Java left later... CS1 CS1 CS2 Prereq: B- Prereq: C CS 139 CS 159 CS 239 CS 240 (4 cr.) (4 cr.) (3 cr.) (3 cr.) CS 149 Students with Not enough for (3 cr.) prior exposure Big-O analysis MA 235 MA 205 (3 cr.) (3 cr.) Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

Key (Research) Questions Effects on retention • Did either change affect retention into CS 159? • Did either change affect retention beyond CS 159? Effects on successful progression • How did they do in CS 240? Effects on underrepresented groups • Do overall effects extend to women and underrepresented minority students? Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

Research Hypotheses CS 159 and CS 240 retention • The split sections improve retention. • The Calculus/B- change led to drop in retention. CS 159 and CS 240 grades • CS 139/149 has no effect on CS 159 or CS 240 grades. • Skip CS 139/149 with AP 4 yields difference in CS 159. • AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. Effect on women and URM students • Split sections improve women/URM retention. • Calculus/B- had disproportionate effect on women/URM. Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

CS 139/149 Demographics 100% 80% 60% 40% 20% 0% Baseline 2013-14 2014-15 White Asian URM Male Female Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

CS 139/149 → CS 159/239 Retention No overall effect after Baseline (pre-2013) splitting sections CS1/1a combined X-sq = 1.81, p = 0.18 Difference between 2013-14 CS1 only CS 139 and previous X-sq = 4.83, p = 0.03* combined CS1a only X-sq = 1.66, p = 0.20 0% 20% 40% 60% 80% 100% combined 2013-14 CS1/1a 2014-15 X-sq = 0.02, p = 0.90 2013-14 CS1 2014-15 No effect after adding X-sq < 0.01, p = 0.95 B- prerequisite 2013-14 CS1a 2014-15 X-sq = 0.04, p = 0.85 0% 20% 40% 60% 80% 100% Retention Attrition Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

CS 159 → 240 Retention No effect after Baseline (pre-2013) accelerated split CS1/1a combined X-sq = 1.97, p = 0.16 2013-14 CS1 only X-sq = 3.33, p = 0.07 CS1a only X-sq = 0.02, p = 0.88 0% 20% 40% 60% 80% 100% combined 2013-14 CS1/1a 2014-15 X-sq = 5.44, p = 0.02* Significant 2013-14 CS1 difference after 2014-15 X-sq = 6.89, p < 0.01** Calculus/B-, but... 2013-14 CS1a 2014-15 X-sq = 0.18, p = 0.67 0% 20% 40% 60% 80% 100% Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

Research Hypotheses CS 159 and CS 240 retention ✕ • The split sections improve retention. ✓ / ✕ • The Calculus/B- change led to drop in retention. CS 159 and CS 240 grades • CS 139/149 has no effect on CS 159 or CS 240 grades. • Skip CS 139/149 with AP 4 yields difference in CS 159. • AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. Effect on women and URM students • Split sections improve women/URM retention. • Calculus/B- had disproportionate effect on women/URM. Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

How were their grades? Linear regression on CS 159 grades • Choice of CS 139 vs CS 149? Significant (p < 0.05) • Grade in CS 139/149? Very significant (p < 0.001) • # of attempts in CS 139/149? Very significant (p < 0.01) CS1.5 Grade CS1.5 grade factor Spearman’s rank coe ffi cient Factor Est. SE t value Pr( > | t | ) CS1 ρ = − 0 . 158 , p = 0 . 009 ** (Intercept) 0.745 0.436 1.708 0.089 . CS1/1a -0.319 0.126 -2.525 0.012 * CS1/1a Grade ρ = 0 . 492 , p < 3 e -16 *** CS1X Grade 0.834 0.096 8.685 < 4 e -16 *** CS1/1a Attempts ρ = − 0 . 182 , p = 0 . 002 ** CS1X Attempts -0.600 0.224 -2.675 0.008 ** Residual standard error: 0.917 on 270 degrees of freedom Multiple R-squared: 0.2633, Adjusted R-squared: 0.2552 F-statistic: 32.17 on 3 and 270 DF, p-value: < 2.2e-16 Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

How were their grades? Linear regression on CS 240 grades • Choice of CS 139 vs CS 149? NOT significant • Grade in CS 159? Very significant (p < 0.001) • # of attempts in CS 139/149? NOT significant − CS2 Grade CS2 grade factor Factor Est. SE t value Pr( > | t | ) CS1 ρ = 0 . 052 , p = 0 . 489 (Intercept) 1.187 0.234 5.079 < 1 e -06 *** CS1.5a Grade ρ = 0 . 467 , p < 5 e -11 *** CS1.5 Grade 0.512 0.075 6.788 < 2 e -10 *** Residual standard error: 0.723 on 177 degrees of freedom Multiple R-squared: 0.2066, Adjusted R-squared: 0.2021 F-statistic: 46.08 on 1 and 177 DF, p-value: 1.651e-10 Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

What about AP students? Average grades in CS 159/240 • AP 4 who skipped CS 139/149 did worse in CS 159 Course Mean Samples Result AP 5, µ = 3 . 4 t = 2 . 4 , d f = 56 . 2 CS1.5/1.5a AP 4 (skip), µ = 2 . 8 p = 0 . 02 * AP 5, µ = 3 . 4 t = 1 . 9 , d f = 53 . 9 CS1.5/1.5a AP 4 (no skip), µ = 3 . 0 p = 0 . 06 AP 4 (skip), µ = 2 . 8 t = − 0 . 7 , d f = 55 . 2 CS1.5/1.5a AP 4 (no skip), µ = 3 . 0 p = 0 . 47 AP 5, µ = 2 . 7 t = 1 . 6 , d f = 44 . 6 CS2 AP 4 (skip), µ = 2 . 2 p = 0 . 12 AP 5, µ = 2 . 7 t = 0 . 2 , d f = 39 . 0 CS2 AP 4 (no skip), µ = 2 . 7 p = 0 . 82 AP 4 (skip), µ = 2 . 2 t = − 1 . 5 , d f = 42 . 2 CS2 AP 4 (no skip), µ = 2 . 7 p = 0 . 15 Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

Research Hypotheses CS 159 and CS 240 retention ✕ • The split sections improve retention. ✓ / ✕ • The Calculus/B- change led to drop in retention. CS 159 and CS 240 grades ✕ / ✓ • CS 139/149 has no effect on CS 159 or CS 240 grades. ✓ • Skip CS 139/149 with AP 4 yields difference in CS 159. • AP 4 + CS 139/149 no difference in CS 159 or CS 240 ✓ relative to AP 5. Effect on women and URM students • Split sections improve women/URM retention. • Calculus/B- had disproportionate effect on women/URM. Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

CS 139/149 → CS 159/239 Retention Baseline (pre-2013) No effect after URM 2013-14 CS1/1a accelerated split X-sq < 0.01, p = 0.98 2013-14 CS1 X-sq < 6e-31, p ≈ 1 Baseline (pre-2013) Women 2013-14 CS1/1a X-sq < 2e-30, p ≈ 1 2013-14 CS1 X-sq = 0.01, p = 0.91 2013-14 CS1/1a URM 0% 20% 40% 60% 80% 100% 2014-15 X-sq = 0.11, p = 0.74 CS1 only 2013-14 URM 2014-15 No effect after X-sq < 0.01, p = 0.96 2013-14 Women CS1/1a increasing to B- 2014-15 X-sq = 0, p ≈ 1 CS1 only 2013-14 Women 2014-15 X-sq = 0, p ≈ 1 0% 20% 40% 60% 80% 100% Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield

CS 159 → CS 240 Retention CS1.5/1.5a Retention After Calculus Change Baseline CS1 (pre-2013) URM 2013-15 CS1/1a combined X-sq < 8e-31, p ≈ 1 Baseline CS1 (pre-2013) Women 2013-14 CS1/1a combined X-sq = 0, p ≈ 1 2014-15 CS1/1a (vs. 2013-14) X-sq = 5.11, p = 0.02* 2013-14 CS1/1a combined Men 2014-15 CS1/1a combined X-sq = 1.15, p = 0.29 Significant drop in women 0% 20% 40% 60% 80% 100% retention after increasing Calculus requirement... Evaluating an Alternative CS1 for Students with Prior Programming Experience SIGCSE 2017 • Kirkpatrick and Mayfield