A Web Based Module A Web Based Module Evaluation System Evaluation - PowerPoint PPT Presentation

A Web Based Module A Web Based Module Evaluation System Evaluation System 1 , Nikki Rochford 1 Nadine Foster 1 , Nikki Rochford 1 Nadine Foster 2 , Stephen Bostock 3 Barry Smalley 2 , Stephen Bostock 3 Barry Smalley Department of Physiotherapy Studies 1 1 , , Department of Physiotherapy Studies 2 , Staff Development and Training 3 School of Chemistry and Physics 2 , Staff Development and Training 3 School of Chemistry and Physics

Objectives of the project Objectives of the project • develop a standard module evaluation form • develop a standard module evaluation form for the BSc (Hons Hons) Physiotherapy ) Physiotherapy for the BSc ( • develop a standard course evaluation form • develop a standard course evaluation form • further develop the supporting software to • further develop the supporting software to analyse and present web- -based summary based summary analyse and present web data data • investigate feasibility of use across campus • investigate feasibility of use across campus

Background Background • Present mechanism of module evaluation • Present mechanism of module evaluation ! Relies on paper questionnaires ! Relies on paper questionnaires ! Effective ! Effective • Problems • Problems ! Time for analysis and production of data ! Time for analysis and production of data ! Questionable cost ! Questionable cost- -effectiveness effectiveness ! Different forms, so not easily compared ! Different forms, so not easily compared

QuestMan QuestMan Questionnaire ionnaire Man Manager ager Quest 1. Production of computer based forms Production of computer based forms 1. 2. Students complete the forms Students complete the forms 2. 3. Data is emailed to special Data is emailed to special Questman Questman account account 3. 4. Collation in a hierarchical directory structure Collation in a hierarchical directory structure 4. 5. Responses are made anonymous Responses are made anonymous 5. 6. Production of results for module leaders Production of results for module leaders 6. ! ! Web pages of analysis and comments Web pages of analysis and comments ! ! Produces “instant” results, when required Produces “instant” results, when required

Example web pages Example web pages • Menu • Menu • A module evaluation form • A module evaluation form • The results • The results ! Data ! Data ! Summary ! Summary ! Comments ! Comments ! Data for Excel ! Data for Excel

Students Evaluation Students Evaluation • To develop the new evaluation form • To develop the new evaluation form ! Small focus group with students ! Small focus group with students ! Pilot of new evaluation form with 1 module ! Pilot of new evaluation form with 1 module • To compare computer with paper system • To compare computer with paper system ! Web or paper for 5 module evaluations ! Web or paper for 5 module evaluations ! Random allocation for comparison ! Random allocation for comparison ! Year 1 and 2 ! Year 1 and 2

Q1 Prefer paper or web web paper 70% 60% 50% 40% 30% 20% 10% 0% paper either web

Q2.Was form design helpful? web paper 60% 50% 40% 30% 20% 10% 0% yes maybe no

Did the medium make a difference? web paper 100% 80% 60% 40% 20% 0% yes maybe no

web paper Q2.4/3.3 Happy to give username? 100% 80% 60% 40% 20% 0% yes maybe no

Q2.3 Need a PC lab session? (web users) 70% 60% 50% 40% 30% 20% 10% 0% yes maybe no

Staff Evaluation Staff Evaluation • Qualitative information • Qualitative information • Staff preference • Staff preference ! Preferred new evaluation form ! Preferred new evaluation form ! 2 staff preferred computer system ! 2 staff preferred computer system ! 3 stated no preference ! 3 stated no preference • Benefits • Benefits ! Time ! Time ! Ease of access of results ! Ease of access of results • Ideas for improvement • Ideas for improvement

Were the evaluation data different Were the evaluation data different from web and paper forms? from web and paper forms? Computerised Paper- -based based Computerised Paper Mode No of students Mode No of students Mode No of students Mode No of students giving a low score, giving a low score giving a low score, giving a low score per question per question per question per question Module 1 4 2.7 (range 0- -22) 22) 4 6.7 (range 0- -31) 31) Module 1 4 2.7 (range 0 4 6.7 (range 0 Module 2 4 7.1 (range 0- -25) 25) 4 5.5 (range 0- -22) 22) Module 2 4 7.1 (range 0 4 5.5 (range 0 Module 3 Module 3 4 4 2.5 (range 0- 2.5 (range 0 -20) 20) 4 4 3.5 (range 0- 3.5 (range 0 -18) 18) Module 4 4 8.5 (range 0- -22) 22) 4 9.7 (range 3- -17) 17) Module 4 4 8.5 (range 0 4 9.7 (range 3 Module 5 4 5.1 (range 0- -14) 14) 3 7.7 (range 1- -15) 15) Module 5 4 5.1 (range 0 3 7.7 (range 1

Were the response rates different Were the response rates different for web and paper forms? for web and paper forms? Number of students completing the evaluations (n=76) Number of students completing the evaluations (n=76) Web % Paper % Web % Paper % (should have been 38 responses ) (should have been 38 responses ) Module 1 36 32 Module 1 36 32 Module 2 Module 2 38 (100%) 38 (100%) 33 33 Module 3 no data no data Module 3 no data no data Module 4 36 17 (45%) Module 4 36 17 (45%) Module 5 Module 5 27 27 25 25 Minimum response rate 45% (paper- -based) based) Minimum response rate 45% (paper Maximum response rate 100% (web form) Maximum response rate 100% (web form)

Further developments Further developments • Standard web form now being used for all • Standard web form now being used for all physiotherapy modules physiotherapy modules ! Years 1, 2 and 3 ! Years 1, 2 and 3 • Standard course evaluation questionnaire • Standard course evaluation questionnaire developed and to be piloted end of May 2002 developed and to be piloted end of May 2002 ! Year 3 ! Year 3 • Providing summary data to fit with • Providing summary data to fit with requirements of the annual course reports requirements of the annual course reports

Conclusions Conclusions • Standard web form • Standard web form ! acceptable to staff and students ! acceptable to staff and students • Summary data • Summary data - - easily interpreted easily interpreted • Feasible for wide use if • Feasible for wide use if ! Students have access ! Students have access ! Simple means provided to set up a new questionnaires ! Simple means provided to set up a new questionnaires