Variable Questions and Small Worlds Variable Questions and Small - PowerPoint PPT Presentation

Variable Questions and Small Worlds Variable Questions and Small Worlds Colin Melville Doug Quinney

Problem Problem Training doctors to become experts ICD-10 disease classification >6500 diseases Training time reducing Hours 1980 approx 70,000 hours 2005 approx 7,000 hours Rotas 1:2 to 1:7 Extended nurse roles

Model of Expertise Model of Expertise >10,000 hours deliberate practice (Ericsson, Road to Excelence) Small worlds (Patel) Experts are only experts within their own small domains

Question Question How can we mimic that prolonged experience? How can we do it cheaply with no danger to patients? Can we give tailored feedback?

Potential Solution Potential Solution Computer-based simulation Text questions +/- media +/- calculations Multiple variables Never tried (to our knowledge) in medicine

Choose area requiring overlearning overlearning Choose area requiring APLS 'Simple' Problems Some calculation Multiple variations on theme Important area

Progress Progress Experimentation with various systems OS Moodle + webwork Proprietary WebCT EDU EDU chosen DQ familiar with it Had all functionality RAD

Questions for development Questions for development How much variation needed? How much variation while preserving realism? What sort of areas best to try out? Any ground rules for further development?

Development Development 6 areas Airway Breathing Circulation Disability Severity scenarios x 12 Arrhythmias x 6

Scenario development Scenario development Each case consists of: Scenario development Written scenario Plausible variables (up to 6) Plausible variations Error checking Encoding into TeX Error checking Encoding into EDU Error checking

Scenario testing Scenario testing Field testing Experts Error checking Incorporating new ideas Novices (not yet done)

Field testing results Field testing results Rating of each question in 3 areas Realism (validity) Accuracy (reliability) Utility (teaching effect) Overall average 4.27/5 Realism 4.3 Accuracy 4.45 Utility 4.1 Comments Much useful on implementation Simpler questions may be most effective Questions requiring memory e.g. of drug doses felt to be trivial Questions requiring application e.g. of scales much appreciated.

Conclusions: the ground rules Conclusions: the ground rules Better to have multiple simpler questions with 2- 3 variables than complex larger question Just as valid, reliable, and useful Much easier in proofing stage Application of knowledge (incl. Calculations) better than recall of details Use of prime numbers minimises repeat questions Simpler methods of implementation needed