Use of Technology in Achievement Levels Setting for NAEP Technology and Engineering Literacy at Grade 8 CCSSO National Conference on Student Assessment June 21, 2016 Steve Fitzpatrick, Ph.D.
Definition of Technology and Engineering Literacy (TEL) “Technology and engineering literacy is the capacity to use, understand, and evaluate technology as well as to understand technological principles and strategies needed to develop solutions and achieve goals.”
TEL Assessment Areas and Practices
TEL Assessment Design The TEL assessment is wholly computer-administered and includes scenario-based and stand-alone items There are three types of scenario-based assessment tasks: long (30 minutes), medium (20 minutes), and short (10 minutes). These scenarios incorporate animations, audio, and video components as part of the TEL items. There are interactive and static stand-alone items.
Technology and Society in TEL
Information and Communication Technology in TEL
Design and Systems in TEL
Interactive Scenario-Based Tasks Iguana Home http://www.nationsreportcard.gov/tel_2014/#tasks/iguana Bike Lanes http://www.nationsreportcard.gov/tel_2014/#tasks/bikelanes
Use of Computers in the ALS Process The interactive nature of the scenarios required that panelists have access to the assessment items in the “live” format in order to make judgments about the items during the item review process. Dual Computer Arrangement • One computer was used to present TEL assessment items to panelists so they could accurately judge the cognitive demand imposed on the student by the items • A second computer was used to complete the ALS activities including presentation of the OIB and collection of panelists’ judgments
Dual Computer Arrangement ALS NAEP NAEP • Take the NAEP TEL • Interact with the items and scenario-based tasks for purposes of reviewing items
Dual Computer Arrangement NAEP ALS • Descriptions of knowledge and skills needed to correctly answer each item • Cut score recommendations for each level at each round • Reviewing feedback data • Responding to questionnaires and evaluations • Selecting exemplar items
TEL ALS Methodology Item mapping methodology • “Modified Bookmark” approach • Adaptation of procedures used for previous NAEP ALS in 2005 • Mathematics (grade 12), 2006 Economics (grade 12), 2009 Science (grades 4, 8, and 12) Conducted using computers with digital materials •
Studies during the ALS Process Meeting Primary Purpose Dates Venue Dual Computer To test the logistics involved in using two laptop December 2-4, 2014 Chandler, AZ Usability Study computers To implement the process designed for the operational Initial Pilot Study March 16-19, 2015 San Antonio, TX meeting and evaluate the need for change(s) To test implementation of modifications based on initial Second Pilot Study June 1-5, 2015 San Antonio, TX pilot study findings To implement achievement levels-setting procedures to September 28 – Operational ALS develop recommendations for consideration of the San Antonio, TX Meeting October 2, 2015 Governing Board
Dual Computer Usability Study - Procedures Five eighth grade science teachers from the Phoenix area participated in the study December 2-5, 2014 All had previous experience in setting performance standards on Arizona tests Each teacher competed a four-hour session – Took a form of the NAEP-TEL assessment – Reviewed the knowledge and skills required by the items – Reviewed the Achievement Level Descriptions and developed borderline performance descriptions – Completed two rounds of standard setting ratings Participants were video taped working through the activities
Dual Computer Usability Study - Findings • Participants were comfortable using the ALS and NAEP computers together and understood the purpose of each • Two participants noted that wheels on the chairs facilitated moving between the two computers • Two participants said that mastering the navigation tabs and terminology was difficult • Directed practice in using the computers should be included during the Pilot Study and ALS meetings
Initial Pilot Study - Procedures Used an Item Mapping Procedure based on the Bookmark Procedure 14 panelists participated Activities took place on two computers • NAEP computer containing test forms, scenarios, and items • ALS computer using web-based standard setting software including digital ordered item books The meeting occurred over a period of three and a half days
Initial Pilot Study - Procedures • Took a form of the TEL assessment on the NAEP computer as a student would • Received training on the NAEP TEL Frameworks and Achievement Level Descriptions • Reviewed TEL scenarios and items using the NAEP computer • Recorded the knowledge and skills assessed by each item using the ALS computer • Performed three Rounds of standard setting ratings using the ALS computer • Recommended Exemplar items for each of the performance levels • Provided feedback throughout the process by responding to questionnaires
Initial Pilot Study – Modifications to Procedures Shorter days – 8:30 to 4:30 Scheduled breaks – 15 min. mid-morning, 30 min. in afternoon More time allotted for the Item Review activity The meeting extended to 4.5 days Re-conceptualized the use of digital technology • Digital Ordered Item List in Excel • Web based bookmark submission and questionnaires • Digital transfer of feedback using secure FTP
Second Pilot Study - Procedures Used an Item Mapping Procedure based on the Bookmark Procedure 29 panelists participated Activities took place on two computers • NAEP computer containing test forms, scenarios, and items • ALS computer using Excel tools, online questionnaires, and FTP site The meeting occurred over a period of four and a half days
Second Pilot Study - Outcomes Participants successfully completed all planned activities The use of EXCEL as the tool for presenting the ordered item list (OIL) worked well Bookmarks and survey responses were collected electronically Feedback to panelists was provided electronically
Operational ALS Meeting September 28 - October 2, 2015 (San Antonio, TX) Item Mapping Procedure based on the Bookmark Method 31 panelists participated Activities took place on two computers • NAEP computer containing test forms, scenarios, and items • ALS computer containing digital ordered item lists The meeting occurred over a period of four and a half days
Digital Tools Used During the ALS Meeting Excel Workbook – A shortcut on the desktop • Review the test • Item Review • Ordered Item List • Interactive Consequences Chart • Exemplar item ratings Survey Monkey – Links in a PDF file • Entering bookmark selections • Questionnaires throughout the process FTP site – A shortcut on the desktop • Round feedback
The ALS Computer Desktop
Excel Workbook • Review the test • Item Review • Ordered Item List • Interactive Consequences Chart • Exemplar tem ratings Excel Workbook Interactive Consequences Chart Exemplar Item Rating
Bookmarks and Questionnaires Survey Monkey • Questionnaires throughout the process • Entering bookmark selections Links to Questionnaires
Questionnaires
Questionnaires
Entering Bookmark Selections
FTP site to transfer feedback FTP Feedback Site
Selected Questionnaire Results Question ALS Average Orientation Topic # Rating End of Day 1 16 How to use the NAEP computer to review items 4.55 How to use the ALS computer to review items End of Day 1 17 4.42 and record comments End of Day 1 19 Comfortable using the two computers together 4.65 Working through the Item Review tab in the End of Day 2 3 4.35 Excel file during the Item Review Successfully navigate through the scenarios on End of Day 2 4 4.65 the two computers. Post Round 1 4 How to select and record bookmarks 3.90 Post Round 2 4 How to select bookmarks 4.58 Post Round 3 4 How to select bookmarks 4.77
Considerations for Digital Implementation Consider standard setting needs early in the test development and implementation process. Allow plenty of time for development and testing of digital standard setting tools. Allow plenty of time for panelists to review complex items during the standard setting process. Consider having an alternate internet source available. Have backup computers available. Have assessment system developers and IT professionals on site.
Recommend
More recommend
