New Perspectives on Estimation and Inference about Research - PowerPoint PPT Presentation

New Perspectives on Estimation and Inference about Research Productivity Cinzia Daraio Department of Computer, Control and Management Engineering A. Ruberti (DIAG) Sapienza University of Rome, Italy E-mail: cinzia.daraio@uniroma1.it Plenary Session: Intriguing Applications of Efficiency Analysis North American Productivity Workshop (NAPW X) Miami Business School, University of Miami Miami, 12 June 2018 Daraio (2018) New Perspectives NAPW X 2018 1 / 40

Motivation Motivation Understanding the functioning of research: How does the production of knowledge work? Assessing research and its impacts: How can we measure productivity/efficiency of research and its impacts? What do we measure? Frascati Manual (OECD, 2015, p. 44-46): “ R&D comprise creative and systematic work undertaken to increase the stock of knowledge and to devise new applications of available knowledge. Criteria: i) novel, ii) creative, iii) uncertain, iv) systematic, and v) transferable and/or reproducible ”. How do we measure it? ( ambiguities of productivity) Why do we measure it? An “intriguing” complex research problem A relevant policy issue. Daraio (2018) New Perspectives NAPW X 2018 2 / 40

Motivation Outline 1 Motivation 2 I: A framework and a performance evaluation model 3 II: Quality and its Impact on Productivity Our Approach to Quality Quality in Higher Education Conditional Efficiency, Separability and Unobserved Heterogeneity A Micro Application: Quality of European Universities 4 III: Inference for Nonparametric Productivity Networks Existing gaps Why tools from the Physics of Complex Systems? A new general production framework A Macro Application: Knowledge Production at Country Level 5 IV: Summary and Conclusions 6 Main References Daraio (2018) New Perspectives NAPW X 2018 3 / 40

I: A framework and a model Introduction and Recent trends The evaluation of research activities is a complex task for many reasons. There are no perfect indicators or metrics which fit for all purposes. Each metric of research assessment is based on a model that can be implicitly or explicitly defined and discussed. If you do not specify your model of the metric you may not check its robustness. changes : In the knowledge production (Gibbons et al. 1994; Nowotny et al. 2001) The crisis of technoscience (Bucchi, 2009) and science (Benessia et al. 2016) Advent of the big data era (the computerization of evaluative informetrics, Moed, 2017) In the Communication of science (Bucchi and Trench, 2014) consequences (see the refs in Daraio, 2018): On the demand and supply side On scholars On the assessment process On the measurement of productivity/efficiency within an assessment process Daraio (2018) New Perspectives NAPW X 2018 4 / 40

I: A framework and a model The need for a Framework Openness INNO EDU RES Theory Knowledge Infrastructures Figure: The ability to develop (understand and use effectively) models for the assessment of research is linked and depends, among other factors, on the degree or depth of the conceptualization and formalization, in an unambiguous way, of the underlying idea of quality. Daraio (2018) New Perspectives NAPW X 2018 5 / 40

I: A framework and a model The Generalized Implementation Problem TRANSLATIONS LOCAL Context of intervention systems analysed at Context of intervention systems LoA (O-committing) Abstraction attributed to analysed at Intellectual Problem Instantiation LoA (O-committing) resources content GLOBAL generates Intervention Theory Method Openness Data generates INNO EDU RES Openness Theory attributed to identifies INNO EDU RES Properties Model (O-committed) Theory identifies Properties Model (O-committed) Figure: An illustration of the generalized implementation problem. Context of Intervention and LoA (Level of Analysis): Left Panel; inclusion of translations: Right Panel. Daraio (2018) New Perspectives NAPW X 2018 6 / 40

I: A framework and a model A Doubly-Conditional Performance Evaluation Model Other Contextual and/or Potential Criteria Environmental Conditions Heterogeneity Rules Time Factors Standards Understandings Incentives Context Actions Consequences Stakeholders - Policy and objectives A I R Input 1 Output 1 C E M Input 2 Output 2 T … … S P Combined/ O U A Transformed R C L S T T …… …… S S Input p Output q Efficiency PROCESSES Effectiveness Figure: A Doubly-Conditional Performance Evaluation Model. Daraio (2018) New Perspectives NAPW X 2018 7 / 40

II: Quality and Productivity II: Quality and its Impact on Productivity From a joint work with L´ eopold Simar and Paul W. Wilson Quality is important but not easy to measure. Quality is linked to productivity and performance but there may be trade-offs between quality and efficiency. The role played by Quality if far from being unambiguously determined. We propose to consider Quality as a latent factor of heterogeneity, that means recognize it is difficult to directly observe it, but it may have an impact on the productivity and performance, although its impact is not a priori known and must be empirically estimated. Powell (1995) in the Strategic Management Journal finds that certain tacit, behavioural, imperfectly imitable features - such as open culture, employee empowerment and executive commitment, can produce advantage; these tacit resources drive the success of TQM . Daraio (2018) New Perspectives NAPW X 2018 8 / 40

II: Quality and Productivity Our Approach to Quality Introduction: Our Approach to Quality Recent works (Bounfour and Edvinsson, 2012) show that measuring and managing the intellectual capital of communities has the potential to change how public sector planning and development is done. Intangibles and intellectual capital have always been considered as relevant factors to the productivity and competitiveness of the private sector as well as of the public sector (Guthrie and Dumay, 2015; Dumay, Guthrie and Puntillo, 2015; Secundo Lombardi, and Dumay, 2018). The measurement of intellectual capital (Bryl, 2018) is an emerging research area in knowledge management (Tiwana, 2000; Alavi and Leidner, 2001 and Liebowitz, 2012). However, being at its infant stage, it still lacks a rigorous methodology for being assessed, as it is the case for managerial quality, that remains difficult to be directly measured and included in a more general performance measurement system. In our approach, Quality is a latent factor of heterogeneity linked to a labour input and/or an intellectual capital input. It may be used in different contexts of manufacturing and public and private service sectors. Daraio (2018) New Perspectives NAPW X 2018 9 / 40

II: Quality and Productivity Our Approach to Quality Our approach to quality and its impact on productivity Nonparametric frontier and efficiency analysis and its robust version : we use the flexible directional distance approach. We apply the new computational methods for directional distances (marginal, conditional and their robust versions) provided in Daraio, Simar and Wilson (2018) with Matlab codes. Conditional frontier models to account for heterogeneity in the production process and analyze the impact of environmental, external factors. We propose the identification of quality as a latent heterogeneity factor linked to some inputs: we use a flexible nonparametric nonseparable model We provide a methodology for identifying quality and analyze its impact on the production process We illustrate the approach through the case of the performance of European Universities. Daraio (2018) New Perspectives NAPW X 2018 10 / 40

II: Quality and Productivity Quality in Higher Education Quality in Higher Education Universities carry out a complex production process. Multiple activities, such as teaching, research and third mission are realized by combining different resources: human capital, financial stocks and infrastructures to produce heterogeneous outputs, such as: undergraduate degrees, PhD degrees, scientific publications, citations, service contracts, patents, spin off and so on, within an heterogeneous environment in which size and subject mix play an important role (e.g. Daraio et al. 2015 JI and Daraio et al. 2015 EJOR and the references cited there). The concept of quality of HEIs is tricky (Sarrico et al. 2010), elusive and complex (Westerheijden et al. 2007) and multidimensional (Blackmur 2007). It’s modeling in quantitative analysis is compelling and challenging (econometric modeling of quality, Daraio, 2017a). In search of Academic Quality (Paradeise and Thoenig, 2015): “Academic quality still remains a black box not only with regard to assessing the outputs, but also in terms of the formal and informal social, cultural and organizational processes adopted by specific university governance regimes”. In this book quality is linked to the academic staff. Daraio (2018) New Perspectives NAPW X 2018 11 / 40