Selection of indicators; the U-Multirank project Frans Kaiser Center for Higher Education Policy Studies ACUP Seminar ‘International university rankings and selection of indicators 21 June 2011 Barcelona
Content Some words on indicators Indicators in U-Multirank • What is U-Multirank? How did we select indicators? • • International orientation • Research Discussion
Indicators What is an indicator? Quantitative • • Proxy of an empirical phenomenon • Not a statistic • ‘norm’-referenced (performance indicator)
Indicators What is a good indicator? • Valid • Reliable Relevant • • Feasible Part of a ‘system’ of indicators •
Indicators
U-Multirank Pilot project • on the design and testing of a new, multidimensional, global transparency instrument • commissioned by the European Commission; 2-year project, starting November 2009 o to allow stakeholders to make informed choices; o to help institutions to position themselves and improve their performance • two phases: o Design of new instrument o Testing the feasibility of new instrument
U-Multirank design principles • Basic epistemological argument : all observations of reality are conceptually driven • Rankings should be based on interests and priorities of its users: the principle of user-driveness • Higher education and research institutions are predominantly multi-purpose: the principle of multi-dimensionality • Higher education and research institutions are generally combinations of different faculties, departments, and programs and show internal diversity: the principle of multi-levelness • Rankings are only useful if institutions/ programs are compared that are sufficiently similar: the principle of comparability • The instrument should refrain from methodological mistakes: the principle of methodological soundness
Design of U-Multirank Literature review
Design of U-Multirank Conceptual framework Stages Enabling Performance Input Process Output Impact Functions & Audiences Functions Teaching & Learning Research context Knowledge Transfer Audiences International Orientation Regional Engagement
Design of U-Multirank Literature review Review of Review of existing rankings existing databases First selection Stakeholder Expert advice consultation Second selection Pre-test Revision Selection for pilot test
International orientation • Many initiatives: no concise, consistent list of indicators • Many rationales • The increasing emphasis on the need to prepare students international labor markets and to increase their international cultural awareness, • The increasing internationalization of curricula • The wish to increase the international position and reputation of higher education and research institutions • Three types of activities Activities to develop and promote international mobility of students • and staff, • Activities to develop and enhance international cooperation, Activities to develop and increase international competition • • In databases mainly input and process oriented
International orientation Focused Institutional Definition Ranking Educational programs in The number of programs offered in a foreign language as a foreign language percentage of the total number of programs offered International academic staff Foreign academic staff members (headcount) as percentage of total number of academic staff members (headcount). Foreign academic staff is academic staff with a foreign nationality, employed by the institution or working on an exchange base International doctorate The number of doctorate degrees awarded to students with a graduation rate foreign nationality, as a percentage of the total number of doctorate degrees awarded International joint research Relative number of research publications that list one or more publications author affiliate addresses in another country relative to research staff Number of joint degree The number of students in joint degree programs with foreign programs university (including integrated period at foreign university) as a percentage of total enrolment
International orientation Field-based Ranking Definition Incoming and outgoing Incoming exchange students as a percentage of total number of students students and the number of students going abroad as a percentage of total number of students enrolled International graduate The number of graduates employed abroad or in an international employment rate organization as a percentage of the total number of graduates employed International academic Percentage of international academic staff in total number of (regular) staff academic staff International research Research grants attained from foreign and international funding bodies grants as a percentage of total income Student satisfaction: Index including the attractiveness of the university’s exchange programs, Internationalization of the attractiveness of the partner universities, the sufficiency of the programs number of exchange places; support and guidance in preparing the stay abroad; financial support; the transfer of credits from exchange university; the integration of the stay abroad into studies (no time loss caused by stay abroad). Joint international Relative number of research publications that list one or more author publications affiliate addresses in another country relative to academic staff Percentage of The number of degree-seeking students with a foreign diploma on international students entrance as percentage of total enrolment in degree programs. Student satisfaction: Rating including several issues: existence of joint degree programs, International orientation inclusion of mandatory stays abroad, international students (degree and of programs exchange), international background of staff and teaching in foreign languages.
Research ‘creative work undertaken on a systematic basis in order to • increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications’ (Frascati manual) • Four main channels for communicating research activities and performance • Text; journal publications, books, • People • Artefacts • Money • Three types of research indicators • Output (count of publications; PhD production) • Outcomes (citation scores, prizes) Impact (effect on society) • • Research versus knowledge transfer
Research Focused Institutional Ranking Definition Expenditure on research The amount of money spent on research activities in the reference year as a percentage of total expenditure Research income from competitive sources Income from European research programs + income from other international competitive research programs + income from research councils + income from privately funded research contracts as a share of total income Research publication output Frequency count of research publications with at least one author address referring to selected institution (within Web of Science) Post-doc positions (share) Number of post-doc positions /fte academic staff Interdisciplinary research activities Share of research publications authored by multiple units from the same institution (based on self-reported data) Field-normalized citation rate Field-normalized citation impact score, where the fields are equivalent to the Thomson Reuters Journal Categories. ‘ Share of highly cited research publications Share of top 10% most highly cited publications; comparing ‘actual’ citation counts to ‘expected’ counts per field Number of art related outputs Count of all relevant research-based tangible outputs in creative arts /fte academic staff Number of international awards and prizes Prizes, medals, awards and scholarships won by won for research work employees for research work and in (inter-) national cultural competitions, including awards granted by academies of science.
Research Field-based Ranking Definition External research income Level of funding attracted by researchers from contracts with external sources, including competitive grants and research income from government, industry, business and community organizations, as a percentage of total income Research publication output Frequency count of (Web of Science) research publications with at least one author address referring to selected institutional unit (relative to fte academic staff) Doctorate productivity Number of completed PhDs per number of Professors (head count)*100 (three-year average) Field normalized citation rate See definition under Institutional Ranking Highly cited research publications See definition under Institutional Ranking
Next phase of U-Multirank • Further development of indicators and indicatorselection • Strong focus on user-driven character • stakeholders’ involvement • User friendliness of webtools • Creation of a larger sample of higher education institutions • Institutionalisation of U-Multirank
Thank you for your attention f.kaiser@utwente.nl www.u-multirank.eu www.u-map.eu
Recommend
More recommend
