Introduction to Evaluation and Assessment August 2019
Office of Assessment, Evaluation and Outcomes • Mission: to create a data-driven culture that promotes growth and improvement of educational programs and individual students. Assistant Dean of Assessment, Tai Lockspeiser, MD, MHPE Evaluation, and Outcomes Director of Office of Assessment, Rachael Tan, PhD Evaluation, and Outcomes Brooke Parsons, MPA Evaluator Susan Peth Sr Evaluation Specialist
Session Goals At the end of this session you will be able to describe and explain… 1. The evaluation system, procedures, and policies 2. Your role and responsibilities in the evaluation process 3. How to access and complete evaluations 4. Different types of assessment that contribute to your grades and how they are administered 5. The exam scoring process and when you’ll receive your results
What is evaluation and why do we do it? • Evaluation involves collecting data about the process and impact of educational experiences on students, faculty, and the institution. • Data is used to support continuous improvement of the educational program and meet standards for accreditation • Multiple sources of data are used
Different types of evaluation data • Surveys in Oasis about quality of curriculum components (i.e. block or course, lecturer, small group facilitator) • End of phase surveys (overall impressions of curriculum and specific measures of attitudes and behavior) • Focus groups • Student assessment data • Faculty input • AAMC questionnaires
Your Responsibility Evaluations are required • Your feedback is critical to the ongoing monitoring, improvement, and accreditation of the medical school • Evidence suggests that failure to complete evaluations in medical school is indicative of professionalism problems in medical school and in practice. • Completing evaluations is a crucial skill to learn and practice in medical school as providing feedback is something done by all practicing physicians
Enforcing Evaluation Requirements • 7 days after course ends – email reminder from Dr. Lockspeiser sent to any students with incomplete evaluations • 14 days after course ends – evaluations all closed and email sent to student about consequences of not completing evaluations • If student doesn't complete evaluations on time for 2 blocks a Professionalism Feedback Form will be completed
We take your feedback seriously • Student feedback has led to numerous changes in the curriculum over past years – Changes to the structure of Human Body Block – Change to pass/fail grading – Introduction of Lecturio • Feedback motivates faculty to improve their teaching and also rewards their teaching effort
Your feedback is confidential • Information provided can be linked to you – This link is ONLY used to monitor completion and quality of feedback • Only members of the office of Assessment, Evaluation, and Outcomes can view this linkage • Data is only reported in the aggregate form • We will always inform you if surveys are anonymous or optional • All comments are available to teachers, block directors, and committee members
Data use not directly related to evaluation • Data Warehouse Governance Committee – Oversees collection, storage, and use of medical student data – Members include faculty and staff from Evaluation, Curriculum, Student Life, Admissions and students • IRB approved studies relating to education must be approved by DWGC • OASIS survey granting permission to use data will be released this week
Characteristics of Good Feedback Specific Descriptive Focused on behaviors Focused on Timely Non-judgmental Improvement
What was the least helpful part of this block for your learning? • “Fragmented and irrelevant” • “Definitely the "parade of lecturers" which is inevitable. I wanted Dr. X, Dr. Y, and the Dr. who did all the connective tissue stuff to teach us more! The other lecturers were fantastic but often continuity of teaching style and material is good for the student and learning.”
Comment on the strengths of this lecturer and ways he/she could improve • “I really appreciated the lecture and it was well done.” • “I appreciated the specific information Dr. X provided, and she welcomed and addressed questions adequately. I also appreciated the examples she provided to help us.” • “She was excellent! I really appreciated her breaking the terminology and logistical things down. Her demeanor and presentation was calming and clear.”
Evaluation process • Evaluations open throughout the block • Due 7 calendar days after being assigned • You are responsible for: – Lecturer evaluations – 2 weeks of Lecturers each block – Block Small Group Facilitator evaluations – All SGFs – Course/Block evaluations – either Essentials Core OR Foundations of Doctoring • Human Body Block – – Everyone completes ALL Lecturer and Facilitator evaluations – Complete either Human Body Block OR Foundations of Doctoring Course Evaluations
Email Notices • You will receive an email notice on the day an evaluation opens, and every Sunday night • OASIS emails contain a summary of the evaluations you have to complete • Use the lower links to click directly to the evaluation without logging in
How to Log into OASIS Use your university credentials to log into: https://ucdenver.oasisscheduling.com/ Mobile device is possible, but works best if you use the individual links from the Oasis notification email
Where to Find Open Evaluations You can also check your progress toward completing your assigned evaluations
Evaluations Currently Open • First Course – Course – Small Groups – Lecturers • Human Body and Foundations of Doctoring – Lecturers
Small Group Facilitator Evaluation Do not evaluate someone you have not worked with
Lecturer Evaluation Did not attend, review video or read handouts OR wrong lecturer
Continue Later vs. Submit • Use “Save but don’t submit, I am not done” until your evaluation is complete. – Encouraged for course evaluations • Evaluation is not complete until you “Submit.”
Assessment in the Essentials Core
Purpose of Assessment • To promote and enhance learning • To inform curriculum and instruction Assessment should not promote competition Criterion-referenced: Everyone can pass! Help each other and you’ll learn more.
Types of Assessment Formative Assessment – Quizzes • Assessment FOR learning • Open-book, collaborative • Completed in Lecturio and Canvas • Small percentage of grade based on participation or performance Summative Assessment – Written Exams • Assessment OF learning • Closed-book, non-collaborative • Completed in Examplify • Large percentage of grade based on performance
Quizzes Lecturio • Required weekly quizzes • Counts for a small percentage of the course grade based on on-time completion • Assessment Office will monitor for genuine effort Canvas • Additional quizzes may be required or optional – varies by Block • May count for a small percentage of the course grade based on participation or performance
Written Exams • Administered in Examplify – Take exams on your own laptop – Download the exam the night before – Proctor provides password on exam day – Secure, timed • Single-best-answer multiple-choice questions • Check syllabus for percent contribution to course grade
Examplify Practice Exam Password: practice1 • Highlight • Calculator Questions? • Strikeout • Notes Contact Matt Cook: MATTHEW.N.COOK@CUANSCHUTZ.EDU • Set timers • Attachments
Exam Day & Beyond • After class takes the exam, an exam validation period begins • Students provided access to questions missed by more than 30% of the class – Posted to Canvas as a quiz – Correct answers provided as well as explanations if available – Students submit a “query” if they believe the identified answer is not correct – Queries submitted directly to the Assessment Team via email: SOM.ExamQuery@ucdenver.edu
Exam Validation • In addition to student review, a psychometric analysis is conducted on the exam – Percentage of students answering a question correctly ( p -value) – Question discrimination – Exam reliability • Identify any questions that do not meet psychometric guidelines and combine with student queries • Block Directors consider content and statistical information to make scoring decisions for the exam
Exam Scoring • Any questions determined to be flawed are removed from the exam before final scoring • Students must achieve a 70% correct or higher to “pass” an exam • Difficulty adjustments made if the exam is tough • Exam scores are released once and they never change
When Will I Receive My Exam Results? Student Queries Final Exam Scores Exam Day Due Released Monday 9am Wednesday 6pm Wednesday Tuesday 9am Thursday 6pm Thursday Wednesday 9am Friday 6pm Friday Thursday 9am Monday 6pm Monday Friday 9am Monday 6pm Monday
Recommend
More recommend
