what do we get from cr and te items
play

What Do We Get From CR and TE Items? Findings from the study The - PowerPoint PPT Presentation

Jun 07, 2023 •754 likes •1.19k views

What Do We Get From CR and TE Items? Findings from the study The Utility of Online Mathematics Constructed-Response Items: Maintaining Important Mathematics in State Assessments and Providing Appropriate Access to Students Phoebe C. Winter, Scott

  1. What Do We Get From CR and TE Items? Findings from the study The Utility of Online Mathematics Constructed-Response Items: Maintaining Important Mathematics in State Assessments and Providing Appropriate Access to Students Phoebe C. Winter, Scott W. Wood, Susan M. Lottridge, Thomas B. Hughes, & Teresa E. Walker Presentation at the National Conference on Student Assessment, June 20, 2013

  2. This work was supported by a Grant for Enhanced Assessment Instruments awarded by the U.S. Department of Education to the North Carolina Department of Public Instruction as lead state, award S368A100003. The opinions in this document are those of the authors and do not necessarily reflect those of the North Carolina Department of Public Instruction, the U.S. Department of Education, or the participating project states. For more information, including sample items and the full report, visit http://www.pacificmetrics.com/omap/

  3. Overview • Background • Overall results – Reliability – Information functions • Item set results – TE vs. MC • Information functions – CR vs. MC • Information functions • Student responses

  4. Purpose • To investigate the feasibility and utility of using mathematics assessments consisting solely of constructed-response (CR) and technology-enhanced (TE) items to measure mathematics knowledge and skills.

  5. Design 4 states • Grade 7 and Algebra I • – Subset of CCSS in mathematics Two core test forms per content area • – Cores designed to measure the same KS – Form 1 consisted of CR/TE items – Form 2 consisted of KS-matched MC items – Same number of points for each core form – Additional items added to include all item formats and balance response time across forms Tests delivered online • All items machine scored • Forms randomly assigned within content area •

  6. Grade 7 Forms N Items on T est Core Form Items CR/TE MC T otal Form 1 CR/TE 13 5 18 Form 2 MC 5 16 21

  7. Algebra I Forms N Items on T est Core Form Items CR/TE MC Total Form 1 CR/TE 14 5 19 Form 2 MC 4 17 21

  8. CR/TE Item Types Graphing items allow the student to draw lines or points (or both) on a grid. Drag-and-Drop items allow the student to match images to their correct counterpart or sort and order components of a response. Constrained Response items have an input tool that pops up when the student clicks in the answer box. The student can either click on the tool buttons or use the matching keys on the keyboard to enter the response. Table Entry items have students use the constrained response tool to enter responses in a table. Free Text Entry items allow the student to use the entire keyboard to respond; restrictions on what the student can enter are removed. Mixed items use both constrained response and free text entry response in different parts of the item.

  9. Research Sample: Schools and Students by State Grade 7 Algebra I Schools Students Schools Students State A 14 1,395 12 746 (31%) (24%) (20%) (22%) State B 13 2,026 18 904 (29%) (34%) (31%) (27%) State C 10 1,134 15 604 (22%) (19%) (25%) (18%) State D 8 1,361 14 1,151 (18%) (23%) (24%) (34%) T otal 45 5,916 59 3,405

  10. Test Information Functions, Grade 7 Form1_Info 10 9 8 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Test Information Functions -- Grade 7 Red = MC Form, Blue = CR/TE Form Blue X = Form 1 Red + = Form 2

  11. Comparison of Item Information by Set, Grade 7 Theta Value KS CR/TE –2 –1 0 1 2 Assessed Type 7SP8b TE      7RP2ac TE    7SP6* ET      7RP3* ET    7EE4a ET     7NS3 ET    7G5* ET      7EE4a* ET    7NS3* ET     7G5 ET      7RP3* ET      7NS3 ET     7G5 ET    

  12. Test Information Functions, Algebra I Form1_Info 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Test Information Functions -- Algebra Blue X = Form 1 Red + = Form 2 Red = MC Form, Blue = CR/TE Form

  13. Test Information Functions, Algebra I with Anomalous Item Set Removed Form1_Info 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Red = MC Form, Blue = CR/TE Form Test Information Functions -- Algebra (Item 17,19 from Form 2, Item 16 from Form 1 Removed) Blue X = Form 1 Red + = Form 2

  14. Comparison of Item Information by Set, Algebra I Theta Value KS CR/TE –2 –1 0 1 2 Assessed Type SID6 GR   FIF4 GR    AREI1 MX    AREI6 TE    FIF4 DD      FLE5 FT     AAPR1 ET      SID6 FT   ACED ET     AREI1 DD    AAPR1 ET    AREI6 ET      FLE5* FT    

  15. Technology Enhanced Items

  16. Algebra I, Set B, TE KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value 0 pt 31% 1 pt 4% 2 pt 65% b -parameter -0.64 point biserial 0.50

  17. Algebra I, Set B, MC1 KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 2% B 3% C 92% D 2% b -parameter –2.17 point biserial 0.20

  18. Algebra I, Set B, MC 2 KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 70% B 9% C 6% D 15% b -parameter -0.61 point biserial 0.25

  19. Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Algebra I Set B Item Information Functions -- Algebra -- Item 4 vs. Item 3 & 5 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

  20. Algebra I, Set J, TE KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution. Statistics Aspect Value 0 pt 31% 1 pt 69% b -parameter -0.70 point biserial 0.49

  21. Algebra I, Set J, MC KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution. Statistics Aspect Value A 49% B 8% C 24% D 19% b -parameter 0.43 point biserial 0.37

  22. Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Algebra I Set J Item Information Functions -- Algebra -- Item 14 vs. Item 18 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

  23. Constructed Response Items

  24. Grade 7, Set B, CR KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx. Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b -parameter 0.39 point biserial 0.57

  25. Grade 7, Set B, MC #1 KS Statement 1: Find a matching value given a proportional relationship and a given value. Statistics Aspect Value A 11% B 4% C 10% D 75% b -parameter –0.89 point biserial 0.37

  26. Grade 7, Set B, MC #2 KS Statement 2: Represent a proportional relation with an equation of type y = kx. Statistics Aspect Value A 83% B 7% C 6% D 5% b -parameter –1.07 point biserial 0.39

  27. Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Grade 7 Set B Item Information Functions -- Grade 7 -- Item 3 vs. Item 2 & 3 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

  28. Grade 7, Set B, CR KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx. Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b -parameter 0.39 point biserial 0.57

  29. Additional Information from CR Item Students use proportional reasoning to find the values, but • do not find the unit rate needed for the expression. Students determine the unit rate for the correct expression, • but do not correctly calculate values. Students do not write an accurate algebraic expression. •

  30. Grade 7, Set G, CR KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Exemplary Response: x = 40 Statistics Aspect Value 0 pt 58% 1 pt 42% b -parameter 0.31 point biserial 0.50

  31. Grade 7, Set G, MC KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Statistics Aspect Value A 10% B 13% C 65% D 13% b -parameter 0.41 point biserial 0.28

  32. Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Grade 7 Set G Item Information Functions -- Grade 7 -- Item 10 vs. Item 13 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

  33. Algebra I, Set G, CR KS Statement: Add and subtract polynomials, combining like terms as needed. Statistics Aspect Value Exemplary 0 pt 67% Response: 1 pt 33% 3 x – 2 y b -parameter 0.62 point biserial 0.51

  34. Algebra I, Set G, MC KS Statement: Add and subtract polynomials, combining like terms as needed. Statistics Aspect Value A 16% B 48% C 14% D 22% b -parameter 0.67 point biserial 0.28

Recommend

HPMS Pavement Data Items 16 different pavement data items are sought (HPMS Field Manual Items

HPMS Pavement Data Items 16 different pavement data items are sought (HPMS Field Manual Items

HPMS Pavement Data Items 16 different pavement data items are sought (HPMS Field Manual Items 47-62) Used in nationwide pavement modeling and cost allocation studies and calculations Pavement Data Input Spreadsheet Item 47 - IRI

362 views • 20 slides
top 3 items for a new NOC top 3 items for a new NOC an informal survey Gareth Eason, HEAnet for

top 3 items for a new NOC top 3 items for a new NOC an informal survey Gareth Eason, HEAnet for

top 3 items for a new NOC top 3 items for a new NOC an informal survey Gareth Eason, HEAnet for TF-NOC, Brussels, 2011-10-12 questions Q1. what, in your opinion, are the first THREE things a new NOC should learn about or put in place to

331 views • 10 slides
New CPH Content Outline Current Exam Coverage General Principles (25 items) Core areas Cross

New CPH Content Outline Current Exam Coverage General Principles (25 items) Core areas Cross

4/3/2017 New CPH Content Outline Current Exam Coverage General Principles (25 items) Core areas Cross Cutting (25 items) Biostatistics (30 items) Communication & Informatics Epidemiology(30 items) Diversity & Culture

444 views • 7 slides
ROCKIN the Ps! State Use Items Why is it mandatory to purchase State Use items? Why is it

ROCKIN the Ps! State Use Items Why is it mandatory to purchase State Use items? Why is it

ROCKIN the Ps! State Use Items Why is it mandatory to purchase State Use items? Why is it mandatory to purchase state use items? As per State Statute K.S.A. 75-3321: Any person or officer authorized to purchase materials and

540 views • 17 slides
Ch 13/14/15: Reduce, Embed, Case News Reduce items and attributes Reducing Items and Attributes

Ch 13/14/15: Reduce, Embed, Case News Reduce items and attributes Reducing Items and Attributes

Ch 13/14/15: Reduce, Embed, Case News Reduce items and attributes Reducing Items and Attributes Reduce Filter Filter Studies presentation days assigned reduce/increase: inverses Items next week papers filter Paper:

330 views • 6 slides
EE&F Action Items Requested EE&F Action Items Requested VSPC recommendations to the

EE&F Action Items Requested EE&F Action Items Requested VSPC recommendations to the

EE&F Action Items Requested EE&F Action Items Requested VSPC recommendations to the Public Service Board: Continue Geotargeting the St. Albans area** Co t ue Geota get g t e St ba s a ea Continue to Geotarget Susie

506 views • 15 slides
Total Line Items Invalid Line Items

Total Line Items Invalid Line Items

Total Line Items Invalid Line Items Deleted Line Items Corrected Line Items 23,633,000 353,000 749,000 129,000 1% 3% 1%

301 views • 27 slides
Total Line Items Invalid Line Items

Total Line Items Invalid Line Items

Total Line Items Invalid Line Items Deleted Line Items Corrected Line Items 38,299,000 430,000 1,155,000 238,000 1.1% 3.0% 0.6% was 1.3% was 3.1%

298 views • 28 slides
1 Action items- specific Action items- specific Western horse mackerel North Sea autumn spawning

1 Action items- specific Action items- specific Western horse mackerel North Sea autumn spawning

Agenda 1) Opening of the meeting 2) Approval of the agenda 3) Follow-up on action items 4) Report from the 2 nd STECF meeting on the landing obligation Articles 15 and 16 of the CFP (section 6 of discard plan) Joint Working Group I and

401 views • 10 slides
Selection of Linking Items Subset of items that maximally reflect the scale information

Selection of Linking Items Subset of items that maximally reflect the scale information

Selection of Linking Items Subset of items that maximally reflect the scale information function Denote the scale information as Linear programming solver (in R, lp_solve 5.5) min(y) Subject to

847 views • 63 slides
T he United States Securities and Exchange deadline for most items was five business days or 15

T he United States Securities and Exchange deadline for most items was five business days or 15

G Corporate Finance Alert March 2004 SEC Adds Eight New Disclosure Items to Form 8-K, Expands Existing Disclosure Items and Shortens the Form 8-K Filing Deadline to Four Business Days By Peter H. Ehrenberg, Esq., Steven M. Skolnick, Esq. and

541 views • 12 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

1.01k views • 49 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

539 views • 18 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

641 views • 33 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

602 views • 37 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

615 views • 40 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

622 views • 28 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

1.18k views • 64 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

419 views • 26 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

358 views • 31 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

801 views • 50 slides
Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items

Table of contents 1. Introduction: You are already an experimentalist 2. Conditions 3. Items Section 1: 4. Ordering items for presentation Design 5. Judgment Tasks 6. Recruiting participants 7. Pre-processing data (if necessary) 8.

800 views • 43 slides
Set-aside Preference and Service Enriched Priority Definitions Common cure items

Set-aside Preference and Service Enriched Priority Definitions Common cure items

Set-aside Preference and Service Enriched Priority Definitions Common cure items Examples Set-aside Preference Common cure items: Draft Referral and Services Agreement Submit with other checklist items Include

556 views • 8 slides
1 01/05/2017 Action items Action items Western horse mackerel Southern horse mackerel

1 01/05/2017 Action items Action items Western horse mackerel Southern horse mackerel

01/05/2017 Agenda 11:30 Opening of the meeting by the chairman, Sean ODonoghue 11:35 Adoption of the agenda 11:40 Follow-up on action items 11:50 Herring in 6.a & 7.b,c Rebuilding plan (presentation & discussion)

290 views • 4 slides

More recommend