Assessment of theses at Assessment of theses at masters and PhD level masters and PhD level Presentation 2 Presentation 2 10 November 2017 10 November 2017 Erasmus+ LOAF Project, Erasmus+ LOAF Project, Vilnius, Lithuania Vilnius, Lithuania Dr Declan Kennedy, Dr Declan Kennedy, Department of Education, Department of Education, University College Cork, Ireland. University College Cork, Ireland. 1 1 1 1
Introduction to Introduction to 1. 1. research theses. research theses. Process for evaluating Process for evaluating 2. 2. research theses. research theses. What do examiners 3. look for when they assess a thesis? Examples of rubrics to 4. help us when assessing theses. 2 2 2 2
Introduction “Thesis student” (“candidate”): any Research Higher Degree student writing a Master’s, PhD or other doctoral thesis or dissertation. “Supervisor”: an academic whose official role is to provide research advice and guidance for a thesis student. “Thesis examiner”: an academic who reads the completed thesis and gives a report recommending a result. 3 3
There are now two basic types of doctoral degrees in addition to the degrees in medicine and law; the PhD (which is the research doctorate and the most common form), and the professional doctorate, of which there are more than 20 different degrees, such as the Doctor of Education (EdD) and the Doctor of Business Administration (DBA) (Nerad 2008). The procedures for assessment of these types of doctoral degrees are basically the same. (Kyvik, 2014) 4 4
Assessment of theses Processes for the assessment of PhD theses and the examination of doctoral candidates differ considerably across countries (Powell and Green 2007) in terms of: The composition of the evaluation committee Whether the examination and defence take place in public or private. Whether the committee can award the degree on the condition of amendments or revisions being made to the thesis. Whether pass or fail is the only alternative. (Powell and Green 2007, Svein, 2014) 5 5
The League of European Research Universities issued recommended guidelines for doctoral training in which the assessment of theses is mentioned (LERU 2007): “The evaluation of the thesis manuscript should be the responsibility of a dissertation committee. The committee should where possible not include the supervisors and must have at least one member from another university, either national or foreign. Final examinations must consist of an in-depth oral discussion with a panel of reputed scholars in which the candidate ‘defends’ her/his thesis.” 6 6
The composition of the evaluation committee The number of members varies between five (the USA), three (Sweden and Norway) and two (Ireland, UK). In the USA, all the members (which constitute the PhD committee) normally come from the same university as the PhD candidate In the UK one of the members must be from another university In Sweden it is often the case, and in Norway it is normally the case, for one of the members to come from abroad. In Lithuania, “the Doctoral Studies Committee forms a Dissertation Defence Board composed of the chairman appointed by the Committee and 4 members.” Note: Examination normally consists of examination of thesis and viva voce (oral examination) . However, in Australia, a thesis examination typically consists of two or three examiners reading the thesis and writing a report recommending a result. In New Zealand, a viva voce or oral examination is used in addition to examiners reading the thesis. 7 7
“ The status of the thesis when submitted for assessment also varies across countries. While in the USA and the UK the examiners may require the candidate to undertake smaller amendments or larger revisions of his or her thesis before its final approval (and publication), in Sweden and Norway the committee must either pass or fail the thesis.” (Kyvik, 2014) 8 8
Examination of theses Most thesis students are mystified by the process of examination (e.g. Johnston, 1997). This is contrary to good assessment practice (e.g. Biggs and Tang, 2011, Brown and Knight, 1994) Often thesis students have no idea what examiners look for in a research masters or doctoral (PhD) theses. Thesis is submitted and then some time later comes back “As supervisors and examiners, we are less mystified by how a written thesis is examined, but much of the examination process is still hidden behind closed doors and I do not have access to what actually happens. My knowledge of what examiners do, like that of most supervisors, is based on personal experience of examining theses, observing how my thesis students were examined, anecdotes about the examination process, and ideal versions of what a thesis should be and what an examiner should do” (Golding et al, 2014) 9 9
“This sort of personal knowledge tends to be partial and incomplete (Holbrook et al. 2007). I have sent thesis students through the examination gauntlet, but is their experience typical? I know how I examine a thesis, but am I typical? I know how a thesis should be examined, but are they actually examined this way? As Denicolo (2003) pointed out, there can be a disturbing disparity between what supervisors expect and what examiners implement. So, what do examiners tend to do as they examine a thesis? Are there any general trends?” - (Golding et al, 2014) 10 10
“Defending” the PhD! The PhD originated in Germany (Simpson, 1983) – the doctoral thesis represented an independent scientific contribution which had to be defended publicly. 11 11
Purpose of viva voce (oral examination) Survey of views (Tinkler and Jackson (2004) The viva allows the examiners to check the candidates’ understanding and ability to produce and present research to PhD standard (36%); It clarifies areas of weakness (32%); It ensures authenticity (31%); It allows the examiners to further develop the candidate’s ideas and to pro- vide advice on publication (25%); It checks that the candidate can ‘defend’ her/his thesis (24%); It enables the examiners to test the candidate on his/her knowledge of the broader literature (22%); It allows the examiners to test the candidates’ oral skills (11%); It can be a site of final decision-making in borderline cases (11%); and It acts as a ‘rite of passage’ (6%). 12 12
The viva (oral examination) Research shows that even when a viva is part of the assessment, most examiners have already made their judgement before the oral presentation on the basis of reading the thesis, and the oral is merely to confirm their assessment (Denicolo, 2003; Jackson and Tinkler 2001, Kyvik 2013; Trafford 2003). Hence, the key aspect of the assessment is what examiners do as they read a thesis (Golding et al., 2014) 13 13
What do examiners look for when they assess a thesis? 14 14
1. Examiners tend to be broadly consistent in assessment of theses Examiner practice is broadly consistent despite differences in institutional instructions (Holbrook et al. 2007), discipline (Bourke, Hattie, and Anderson 2004; Lovat, Holbrook, and Bourke 2008), nationality (Pitkethly and Prosser 1995), level of degree (Bourke and Holbrook 2013), or the experience of the examiner (Kiley and Mullins 2004). 15 15
Where a number of examiners are involved in assessing a thesis, Holbrook et al. (2008) found examiners gave consistent recommendations for 96% of the 804 theses they studied. An inconsistency was defined as one or more examiners recommending accept or accept with minor revisions, while one or more examiners recommended revise and resubmit or fail. Even when the recommendations were inconsistent, the content of their reports, what the examiners wrote about, tended to be the same (Holbrook et al. 2008). Inconsistent examiners tend to offer consistent recommendations if they have a chance to negotiate in a thesis examination committee or after viewing the oral presentation of a thesis student. 16 16
“Because it is rare for examiners to give wildly inconsistent recommendations, you also need not worry about ‘the examiner from hell’ who gives a radically divergent judgement from the other examiners, or who recommends your thesis fail for a seemingly crazy reason. These are rare exceptions, memorable only because they are extreme. All your examiners are likely to be looking for similar qualities in your thesis, and they are all likely to give a consistent recommendation to pass (though they may disagree about whether minor or major changes are needed) (Holbrook, Bourke, Lovat, & Fairbairn, 2008; Lovitts, 2007).” (Golding, 2017) 17 17
2. Examiners expect a thesis to pass Examiners begin reading with curiosity and enthusiasm, expecting a thesis to be good and ‘hoping to find their task rewarding and enjoyable’ (Johnston 1997). They know years of effort has gone into a thesis, and it has been judged worthy by supervisors (or at least passable); so, they anticipate it will pass, and even want it to pass (Kiley and Mullins 2004). 18 18
Examiners are reluctant to fail a thesis. They consider a thesis a pass until it is proven a fail, and will only consider failing a thesis if there are significant errors or omissions that threaten the credibility of the research (Holbrook et al. 2004). Even then, examiners can get upset if they have to recommend a fail or even a resubmit. Less than 1% of examiners recommend a fail (Lovat, Holbrook, and Bourke 2008). 19 19
Recommend
More recommend
