Remote invigilation: a spotlight on assessment administration Nicola Mellor, Qualifications Director, CII Liam Simington, Assessment Manager, CII
Standards. Professionalism. Trust.
Existing research • Key themes and names • Ref to reading list Bibliography available Standards. Professionalism. Trust.
Existing research weaknesses Operational focus Lack of Commercial published sensitivity research Reputational risk "keen to protect their own reputations for secure and safe assessment systems" (Jefferies, 2017) Standards. Professionalism. Trust.
Existing research weaknesses Operational Small sample focus questionnaires Lack of Research issues Technical Commercial published (method and variation in RI sensitivity research focus) Exam room as Reputational risk gold standard Standards. Professionalism. Trust.
Pilot 2 providers 27 candidates 2 settings (office and home) 4 countries Standards. Professionalism. Trust.
How would you rate RI Pilot survey for convenience? 1. VERY CONVENIENT 7 2. CONVENIENT 2 Where do you 3. NEUTRAL 0 prefer your 4. INCONVENIENT 0 invigilator to be … 5. VERY INCONVENIENT 0 0 2 4 6 8 1. REMOTE 4 2. NEUTRAL 4 Did your exam start.. 3. IN THE ROOM 1 0 2 4 6 VERY LATE? 1 ..SLIGHTLY LATE? 7 ..ON TIME? 1 ..EARLY? 0 ..VERY EARLY? 0 0 2 4 6 8 Did your exam start.. Standards. Professionalism. Trust.
Post pilot questionnaire UK preference Non-UK 505, preference 15% 5, 1% 2775, 85% 442, RI Centre RI Centre 99% Non-UK fee will UK- fee will pay pay 210, 7% 472, 36, 8% 48, 14% 10% 2598, 383, 79% 82% Below proposed Propsed Above proposed Proposed Above proposed Below proposed Standards. Professionalism. Trust.
Further research wish list 3. Cheating behaviours 1. Human interactions 2. Environmental factors and risk perception
The watching eye Standards. Professionalism. Trust.
The neglected situation ‘Managing the Quality of Data Collection in Large Scale Assessments’. (Maddox, B, 2017, p8), OECD Headquarters, Paris. 11- 12 May 2017. Accessed online https://www.oecd. org/skills/piaac/Interview er- Respondent%20Interacti ons%20in%20PIAAC_Br yan%20Maddox.pdf 25. 10.18 Standards. Professionalism. Trust.
Dynamics of control vs. validity The Authenticity Square H Authenticity of task to construct The sweet spot: High confidence L H Authenticity of candidate performance Standards. Professionalism. Trust.
Dynamics of control vs. validity The Authenticity Square H Authenticity of task to construct The sweet spot: High confidence Trend: the greater the control, the more the task authenticity is challenged L H Authenticity of candidate performance Standards. Professionalism. Trust.
Who manages the risks and controls? Candidate Awarding organisation - - Invigilation •- Travel - Rules •- Conduct - Environment (physical) - Environment (technical – equipment, Centre internet and platform) - Who is in control? How does this affect the way we perceive risk? Factors: anxiety disorders; access arrangements; unconscious biases in risk perception Standards. Professionalism. Trust.
Who manages the risks and controls? Candidate Awarding organisation - - Invigilation •- Travel - Rules •- Conduct - Environment (physical) - Environment (technical – equipment, Centre internet and platform) Who is in control? How does this affect the way we perceive risk? - Conduct - Invigilation R.I. - Environment (physical) - Rules - Environment (technical – internet - Environment (technical – platform) and equipment) Factors: anxiety disorders; access arrangements; unconscious biases in risk perception Standards. Professionalism. Trust.
The interaction between cheating and RI • Usual exam nerves • Pressure of being watched Factors: • Cultural context • Communications and marketing • Adverse publicity • Choice and communication of model The 'fraud triangle’. • Penalties and (Jefferies, 2017,after disciplinary Ramos, 2003) processes Standards. Professionalism. Trust.
The interaction between cheating and RI • Hide unauthorised • Usual exam nerves materials or web access • Pressure of being • Assume an invisible watched invigilator is not Factors: watching / assume the • Cultural context invigilator is watching • Communications closely at all times and marketing • Adverse publicity • Choice and communication of model • Penalties and disciplinary processes Standards. Professionalism. Trust.
The interaction between cheating and RI • Hide unauthorised • Usual exam nerves materials or web access • Pressure of being • Assume an invisible watched invigilator is not watching / Factors: assume the invigilator is • Cultural context watching closely at all • Communications times and marketing • Adverse publicity • Choice and communication of model • Publicity of online cheating • Penalties and • But cannot see it happening disciplinary directly processes Standards. Professionalism. Trust.
Detection and prevention This Photo by Unknown Author is licensed under CC BY-SA Standards. Professionalism. Trust.
Vision of the R.I. future This Photo by Unknown Author is licensed under CC BY-SA This Photo by Unknown Author is licensed under CC BY-SA-NC Standards. Professionalism. Trust.
Recommend
More recommend
