Implementation of large-scale online mathematics examinations: successes, challenges and thoughts for the future Martyn Parker and Lynne Bailey, University of Warwick
Outline Operating context Assessment design Implementation Successes, challenges and thoughts for the future
Caveat • Cannot cover every point around the process. • Won’t look at dead ends.
Outline Operating context Assessment design Implementation Successes, challenges and thoughts for the future Thoughts for the Future
Operating context Over 2000 mathematical sciences students Maths + Discrete Mathematics MORSE Data Science Stats/Physics… Mathematics Typical assessment 10% coursework 90% examination University year Term 1 (Oct - Dec) Term 2 (Jan - Mar) Term 3 (Apr - Jun)
Operating context Over 2000 mathematical sciences students Maths + Discrete Mathematics MORSE Data Science Stats/Physics… Mathematics Typical assessment 10% coursework 90% examination University year Term 1 (Oct - Dec) Term 2 (Jan - Mar) Term 3 (Apr - Jun) 13th Mar 20th Apr
Operating context 13th March • All assessments and exams will be online with immediate e ff ect. • No first year examinations • All planned centrally timetabled written examinations replaced by online assessment. Assignments Open book assessment Files-based open book assessment Multiple Choice Questions Spoken exam under timed conditions
Operating context 13th March • All assessments and exams will be online with immediate e ff ect. • No first year examinations • All planned centrally timetabled written examinations replaced by online assessment. Assignments Open book assessment Files-based open book assessment Multiple Choice Questions Spoken exam under timed conditions
Outline Operating context Assessment design Implementation Successes, challenges and thoughts for the future Thoughts for the Future
Assessment design Constraints Time Five weeks until first sitting. Academic rigour Need to maintain external and internal quality assurance processes. Exams already written, moderated and approved by externals. Accreditating One body provides exemption from professional examinations. bodies Technology GDPR; information security Students Located all over the world; range of di ff erent environments; study/assessment expectations.
Assessment design Constraints Time Five weeks until first sitting. Academic rigour Need to maintain external and internal quality assurance processes. Exams already written, moderated and approved by externals. Accreditating One body provides exemption from professional examinations. bodies Technology GDPR; information security Students Located all over the world; range of di ff erent environments; study/assessment expectations.
Assessment design “If I had to reduce all of educational psychology to just one principle, I would say this: the most important single factor influencing learning is what the learner already knows.” Ausubel, D., Novak, J., & Hanesian, H. (1978). Educational psychology: A cognitive view (2nd ed.). New York: Holt, Rinehart & Winston.
Assessment design Evoked prior experiences Evoked conception of How the student conceive of learning in the specific context of their study learning Evoked Those motivational aspects evoked by the context motivation Evoked self- e ffi cacy Confidence students have in their ability to succeed in a specific situation Paul Ashwin & Keith Trigwell (2012) Evoked prior experiences in first-year university student learning, Higher Education Research & Development, 31:4, 449-463, DOI: 10.1080/07294360.2011.634384
Assessment design Student’s situation Student’s evoked prior experience Prior Overall academic Student’s perception of their situation outcomes experience Student’s approaches to learning Student’s outcomes Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education. Buckingham, UK: Society for Research into Higher Education and Open University Press.
Assessment design Student’s situation On-campus Student’s evoked prior experience Social contact Student’s perception of their situation Expecting and preparing for Student’s approaches to learning closed book time-constrained Student’s outcomes written exams with known rubric Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education. Buckingham, UK: Society for Research into Higher Education and Open University Press.
Assessment design Create a process that replicated the students’ prior experiences and their current approach. Files-based open book exams Minor updates to current papers: de-emphasise the importance of “bookwork” Use tested technology Work with the students on their assessment experience Develop literacy around any changes
Assessment design Create a process that replicated the students’ prior experiences and their current approach. Files-based open book exams Minor updates to current papers: de-emphasise the importance of “bookwork” Use tested technology Work with the students on their assessment experience Develop literacy around any changes
Outline Operating context Assessment design Implementation Successes, challenges and thoughts for the future Thoughts for the Future
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Feedback 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims Technology 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation O ffi ce Lens OneDrive Genius Scan Adobe Scan
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims Recruited Technology 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims Recruited Technology Initial feedback 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation Practice exam Draft instructions Complete the paper Submit the paper Feedback
Implementation
Implementation
Implementation
Implementation
Implementation
Implementation
Implementation
Implementation
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims Recruited Technology Initial feedback 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation
Implementation
Implementation • 24-hour window* • Assessments were timed (e.g. 2 hours + extra time if applicable) • 45 minutes to prepare submission. • Random sample selected for post exam: “Sampling Conversations” * there are some exceptions to this
Implementation
Implementation
Implementation
Implementation Mon Tue Wed Thu Fri Sat Sun 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Aims Tech. Demo Recruited Initial feedback 23 24 25 26 27 28 29 30 31 1 1 2 3 4 5 2 3 4 5 6 7 8 9 10 11 12 Practice 13 14 15 16 17 18 19 Practice Feedback deadline 20 21 22 23 24 25 26 Exams start 27 28 29 30
Implementation
Implementation
Implementation
Implementation Over 1,500 students
Implementation
Implementation
Implementation 10% 366 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 0
Implementation 163 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 0
Recommend
More recommend
