PARCC’s Use of Technology in Performance Level Setting CCSSO National Conference on Student Assessment June 21, 2016 Eric L. Moyer, Ph.D. Laurie Laughlin Davis, Ph.D.
PARCC Online Testing • The PARCC tests are administered online, although paper-and-pencil versions are available for schools that are not ready for the computer-based version. • PARCC’s ultimate goal is the digital delivery of the PARCC ELA and mathematics assessments on the widest variety of devices that will support interchangeable scores • Technology enhanced items and online delivery of tests require a new approach to managing standard setting processes, materials, and logistics 2
Example Items from PARCC Practice Sets 3
Example Items from PARCC Practice Sets 4
Example Items from PARCC Practice Sets 5
Example Items from PARCC Practice Sets 6
Evidence Based Standard Setting (EBSS) • Integrates empirical data from systematic research with content expert judgment and policy goals in setting performance standards for students Students Policy Content Research 7
Steps in the EBSS Approach 1. Define outcomes of interest and policy goals 2. Develop research, data collection, and analysis plans 3. Synthesize the research results 4. Conduct pre-policy meeting 5. Conduct PLS meetings with panels 6. Conduct reasonableness review with post-policy panel 7. Continue to gather evidence in support of standards (longitudinal research planned for 2015-2017) 8
Extended Yes/No Judgment Task (Modified Angoff) Panelists made judgments for each item and • performance level Dichotomous: “ Would a borderline Level 4 student be likely* to – get this item correct?” Response: ‘Yes’ or ‘No’, translated to 1 point/0 point Polytomous: “How many points would a borderline Level 4 – student likely earn on this item (or set of items)?” Response: 0 points to maximum possible points Points are summed to get total score • 9
PARCC Uses of Technology in Performance Level Setting 1. Post-secondary Educators’ Judgment Study An entirely virtual process resembling a standard setting but – with nearly 200 participants in locations across the country 2. Meeting Facilitator Training An entirely virtual process used for training 3 dozen staff – members across multiple locations 3. Dry-Run of Performance Level Setting A hybrid virtual (for pre-work) and in person process – conducted with approximately 20 educators 4. Performance Level Setting Meetings A hybrid virtual (for pre-work) and in person process – conducted with approximately 200 educators 10
What is Moodle? https://moodle.org/ 11
What is Moodle? Secure access using unique login id and password • Access to external links; view external resources and • access the secure TestNav 8 site for viewing PARCC online test forms Data collection using online surveys for item judgments, • evaluation, panelist information, etc. Document storage for panelist and facilitator access • Discussion forum for panelists discussion and • communication with facilitators for answering questions 12
Postsecondary Educators’ Judgment Study (PEJ)
Overview of PEJ Study • Purpose: – Minimum level of performance that would indicate students are academically ready to succeed in entry-level, college- credit bearing postsecondary courses – Results used to inform next steps in performance level setting process • Online standard setting process – All virtual process – Only 1 round – Extended Yes/No Modified Angoff Judgment • Nearly 200 faculty from various institutes of higher education 14
Use of Moodle for … PEJ Study • Accessed training and reference materials • Completed demographic questionnaires • Reviewed PARCC items through link to TestNav 8 • Made judgments about PARCC items • Posted questions to study facilitators and other participants • Completed an evaluation survey at the end of the process 15
Survey Results for PEJ Study— ELA/L Grade 11 Not Partially Very Successful Successful Successful Successful Statement Use of the website for the 3% 12% 54% 30% study Use of online training for 1% 17% 50% 32% the study process Use of TestNav to review 4% 19% 48% 29% test items Use of online survey to 3% 19% 44% 33% collect Item judgments 16
Survey Results for PEJ Study— Algebra II Not Partially Very Successful Successful Successful Successful Statement Use of the website for the 1% 7% 57% 35% study Use of online training for 2% 11% 52% 35% the study process Use of TestNav to review 0% 8% 56% 36% test items Use of online survey to 0% 12% 51% 37% collect Item judgments 17
Training of Meeting Facilitators
Training of Meeting Facilitators • Provide in-depth training – Live and “on-demand” training • 10 training sessions • General and specific topics – Track facilitator completion and retention of training – Build facilitator familiarity with Moodle • Participants: – 36 content and psychometric facilitators and analysts – Multiple locations and two vendors 19
Use of Moodle for … Facilitator Training • Training in the Moodle Site – Access to training materials – Moodle Bootcamp • Modelling of Moodle Site – Demonstration of tasks – “Switch role to…” • Practice with Moodle Site – Facilitators “own” the site – Built into the presentation slides 20
Dry-Run of Performance Level Setting
Dry Run Meeting: Purpose and Goals • Purpose: – Evaluation of the process due to the innovative features of PARCC assessments – Collect panelist feedback related to strengths and weaknesses of process and methodology – Identify areas in which updates in process should be made (timing, process, materials, etc.) • Participants: – 2 Meetings: Grade 11 ELA/L and Algebra II – About 10 panelists per meeting 22
Use of Moodle for … Dry Run Meetings • Access to pre-work; completed before dry run meeting (tutorials, practice tests, demographic questionnaire) • “Experienced the Test” through link to test items • Collaborated with team to create borderline PLDs through links to documents on Google drive • Reviewed items and recorded item judgments (practice round and 3 judgment rounds) • Completed process evaluation surveys throughout the process 23
Dry-Run “Lessons Learned” Positive “Lessons” • Moodle provided control over secure materials while also providing a single point of sign-on for all meeting activities • Panelist data entry of their item judgments resulted in faster turnaround of results • Facilitators could monitor panelists access and completion of tasks • Panelists reported that they liked the use of Moodle • Quick resolution of issues that arose during meeting 24
Technology Take-Aways From the Dry-Run Learned “Lessons” • Use of “conditional” logic, which hides content within the site until prior activities are completed, resulted in some panelist frustration – Result: conditional logic was eliminated for all tasks except for the non-disclosure agreement • Data entry of item judgments was considered inefficient and laborious – Result: Item judgment process and survey were revised. 25
Original Data Entry Form 26
Revised Data Entry Form • After you have recorded your judgments on paper you will then transfer them to Moodle 27
Technology Take-Aways From the Dry-Run Learned “Lessons” • Technology requirements must be met – Internet connectivity at meeting site is critical – Have onsite technical support from both the hotel as well as from your vendor – Moodle server capacity needs to be sufficient for the number of simultaneous users • Not a completely online process, panelists needed to manage multiple resources 28
Performance Level Setting Meetings
Use of Moodle for… Performance Level Setting Participants • Access to pre-work; completed before PLS meeting (tutorials, practice tests, demographic questionnaire) • “Experienced the Test” through link to test items • Collaborated with team to create borderline PLDs through links to documents on Google drive • Reviewed items and recorded item judgments (practice round and 3 judgment rounds) • Completed process evaluation surveys at end of process 30
Use of Moodle for … Performance Level Setting Project managers/project leads • Monitored panelist completion of pre-work activity prior to the onsite meeting Facilitators • “Modelled the Moodle” at the introduction of each task to orient panelists to tasks and materials • Received feedback data following each judgment round in the facilitator section of each Moodle site Data Analysts • Extracted data for analysis; posted feedback data 31
Use of Google Documents Development and Review of Borderline PLDs • Original PLDs stored and accessed through Moodle • Specific Google document was assigned and accessed by each group to review and revise borderline PLDs – Google documents allowed real-time access and editing of the same document by multiple members of a group – Facilitators had access to all table group documents and created a master version for full group review • Improved efficiency, accuracy, and access to 32 borderline PLDs.
Recommend
More recommend
