Through Feature Analysis Eva L. Baker, Ayesha Madni, Jenny Kao, - PowerPoint PPT Presentation

Career Readiness Inferences from High School Assessments Through Feature Analysis Eva L. Baker, Ayesha Madni, Jenny Kao, Noelle Griffin, Kilchan Choi, & Li Cai NCSA Jun 28, 2018

Overview • Can existing examinations be re-purposed as a measure of workforce readiness? • Strategy - using Feature Analysis as a technique to identify relevant item elements and confirm them by analyzing student test data • Summary of process and descriptive results for items drawn from Smarter Balanced Assessment Consortium (SB), ELA & Math tests • International (South Korea) benchmark analysis 2

In the EAG Project • Goal was to identify features related to career readiness in the 8th and 11th grade SB English Language Arts (ELA) and Math tests • Career readiness features were drawn from the Department of Labor O*NET list as well as from prior CRESST work • O*NET includes skills, abilities, and work activities for various occupations • https://www.onetonline.org 3

Feature Analysis • Feature Analysis (FA): the systematic rating of test items using attributes compared with test results • Intended to identify attributes that contribute to performance or predict learning • Prior research applied FA to state assessments, Smarter Balanced Assessment Consortium tests, Navy examinations, and PBS Kids games. • Earlier efforts used categories of cognition, content, linguistics, and task features. 4

5

6

7

Feature Rating • Qualitative rating of items against a set of attributes followed by a subsequent quantitative analysis to determine item performance. • A subset of SB items were selected for ELA and math from each of the three cut-scores considering claims, targets, and item types across grades 8 and 11 and years 2015 and 2016 (320 ELA items and 280 math items) • Career ready features informed by ontologies, extensive resource analysis for EMT and web developer domains, and expert review and analysis • Career readiness features categorized into skills, abilities, and work activities as defined by O*NET 8

Some Research Questions 1. What particular attributes/features do the items contain? 2. Which features appear more frequently across items? 3. Are there differences in feature representation across domain areas? 4. What particular attributes/features of items explain variations in item characteristics such as increased or reduced difficulty across items and why? 5. What features are most important to elicit and assess key readiness outcomes? 9

ELA Feature Analysis • Almost all of the ELA items have only one “skill” feature present: ELA Results: reading comprehension or active listening 126 ELA items for 8 th grade • • No item was rated as requiring rated complex problem solving, critical 138 ELA items for 11 th grade • thinking, or judgment and rated decision making • 152 of these items • With the exception of a few administered in 2015 features, overall low feature • 158 items administered in frequency 2016 • About half of the items were rated as requiring 3 or 4 ability features 10

Career-Readiness Features for ELA Skills Abilities Work Activities & Context (7 features) (12 features) (11 features) Active learning (0%) Deductive reasoning (79.2%) Analyzing data & information (7.6%) Active listening (21.6%) Flexibility of closure (12.1%) Documenting/recording info (8.7%) Complex problem solving (0%) Fluency of ideas (8.3%) Getting information (5.3%) Critical thinking (0.4%) Inductive reasoning (43.2%) Identifying objects, actions, & events (9.8%) Judgment & decision making (0%) Information ordering (8.7%) Importance of being exact or accurate (100%) Monitoring (0%) Memorization (0%) Interacting with computers (0.4%) Reading comprehension (72.0%) Oral comprehension (21.6%) Judging the qualities of things, services, or people (1.5%) Problem sensitivity (0%) Making decisions and solving problems (2.3%) Time sharing (84.5%) Organizing, planning, and prioritizing work (0%) Written comprehension (100%) Processing information (0%) Written expression (8.3%) Thinking creatively (1.1%) Visualization (0%) 11

ELA Feature Correlations • As expected, active learning negatively correlated with reading comprehension • Fluency of ideas strong positive correlation with documenting/recording information (0.98) • Information ordering, analyzing data and information, and fluency of ideas highly correlated • Smaller positive correlation between inductive and deductive reasoning 12

Math Feature Analysis • Reading comprehension predominant skill • Limited representation of Math Results: additional skills - 12% of items 65 Math items for 8 th grade • contained critical thinking and 25% required complex problem rated solving 121 Math items for 11 th grade • • Deductive reasoning – “ability” rated feature required by 97% of the • 145 of these items items administered in 2015 • Inductive reasoning rated for • 41 items administered in 2016 35% of the items • Math reasoning was required in 58% of the items and number facility was required in 73% of the items 13

Career-Readiness Features for Math Skills Abilities Work Activities & Context (4 features) (10 features) (8 features) Active learning (4.8%) Deductive reasoning (97.3%) Analyzing data and information (77.4%) Estimating the quantifiable characteristics of Complex problem solving (25.3%) Flexibility of closure (3.2%) products, events, or information (21.0%) Critical thinking (11.8%) Inductive reasoning (35.5%) Getting information (25.8%) Identifying objects, actions, and events Reading comprehension (76.3%) Information ordering (2.7%) (48.4%) Mathematical reasoning (58.1%) Importance of being exact or accurate (71.5%) Making decisions and solving problems Memorization (0%) (22.0%) Organizing, planning, and prioritizing work Number facility (72.6%) (34.4%) Time sharing (5.9%) Processing information (67.7%) Written expression (1.1%) Visualization (12.4%) 14

Math Feature Correlations • Medium positive correlation between complex problem solving and inductive reasoning • Small to medium positive correlation between critical thinking and organizing, planning, and prioritizing work • Medium positive correlation between reading comprehension and analyzing data and information 15

Take-Away from Descriptive Information • Smarter Balanced: • “Assessment is relevant to college and career.” • “Measure critical thinking and problem solving skills students need for success.” • Most predominant – reading comprehension • Limited presence - critical thinking, problem solving • Deductive and inductive reasoning are present across both math and ELA • Lower frequency for features that may apply across different careers 16

More Statistical Modeling To examine the relationship between the career readiness feature ratings with item level psychometric characteristics – in the present case item difficulty parameter. 17

WLS with Known Weights Complexity with adaptive test Weighted Least Squares (WLS) estimator: 2 are known error 2 , where 𝜏 1 2 , 𝜏 2 2 , … , 𝜏 𝑜 • Var ( 𝑍 𝑗 ) = 𝜏 𝑗 variance from calibration 2 = variance of item difficulty parameter for item i • 𝜏 𝑗 2 , thus, items with larger error • Weight ( w) = 1/ 𝜏 𝑗 variance put less weights in regression fitting • Variance of item difficulty = square of standard errors of item difficulty / N of items in the item pool 18

8 th Grade ELA ( 𝑆 2 = 0.56) Feature Estimate SE t p Intercept 1.028 0.259 3.97 0.000 Fluency of ideas* 1.233 0.335 3.68 0.000 Deductive reasoning 0.215 0.220 0.98 0.330 Flexibility of closure 0.116 0.162 0.71 0.477 Analyzing data or information 0.010 0.691 0.01 0.989 Identifying objects, actions, and events -0.002 0.692 0.00 0.998 Getting information -0.030 0.498 -0.06 0.952 Timesharing -0.051 0.145 -0.35 0.727 Inductive reasoning -0.070 0.187 -0.38 0.708 Reading comprehension* -0.568 0.198 -2.87 0.005 Active listening* -1.180 0.249 -4.74 <.0001 19

11 th Grade ELA ( 𝑆 2 = 0.54) Feature Estimate SE t p Intercept 0.558 0.296 1.88 0.062 Fluency of ideas 1.266 0.680 1.86 0.065 Identifying objects, actions, and events* 1.219 0.435 2.8 0.006 Making decisions and solving problems 0.848 0.777 1.09 0.277 Deductive reasoning* 0.659 0.141 4.68 <.0001 Timesharing 0.079 0.170 0.47 0.641 Written expression 0.075 0.434 0.17 0.863 Getting information 0.056 0.364 0.15 0.878 Inductive reasoning -0.048 0.139 -0.34 0.732 Active listening -0.321 0.344 -0.93 0.353 Reading comprehension -0.331 0.328 -1.01 0.314 Flexibility of closure* -0.961 0.125 -7.67 <.0001 Information ordering -1.265 0.911 -1.39 0.167 20