Comparative Effectiveness Evaluation and Monitoring, Austrian perspectives Workshop: More Effective Public Workforce Programs through Comparative Performance Monitoring Helmut Mahringer November 13, 2018
Impact evaluation on active labor market policies in Austria Improvement of evaluation culture in Austrian Labor Market Policy Since Austria joined the EU in 1995 Initially: Obligation for systematic evaluation of European Social Fund (ESF) defined by European Commission (EC) First large project commissioned by the ML: Evaluation of Austrian ESF-Program compared treatment effects of training between nationally financed programs and ESF (OP 2000-2006) Application of monitoring tools and counterfactual impact analysis Mutual trust between Public Employment Service (PES) and its supervisory authority (the Federal Ministry of Labor (ML)) that evaluation results are used in a constructive way PES increasingly engaged in monitoring and impact analysis (e.g. BSC , randomized controlled trial on case loads of counselling officers) Development of data-background by PES and ML: European best practice example 2003 Based on administrative employer-employee microdata joined with PES data on unemployment, UI-benefits, active LMP measures (EU best practice example) Evidence from counterfactual impact evaluation available for most workforce development programs in Austria 1
Evidence from counterfactual impact analysis of LMP in Austria Skills and training measures Job-search training Causal evaluation strategy: control group design (propensity score matching) No significant effect on outcome variables: (unsubsidized) employment Occupational orientation and guidance Causal evaluation strategy: control group design (ps matching) No significant effect on (unsubsidized) employment Basic skills development and further training Causal evaluation strategy: control group design (ps matching) Positive effects on employment (after lock-in) Higher impact for women, middle age and higher qualified 2
Evidence from counterfactual impact analysis of LMP in Austria Employment measures Temporary wage subsidies: Causal evaluation strategy: exploiting variation in regional treatment intensities combined with control group design (matching unemployed and jobs) High deadweight effects (~50% of the cases), positive effects on employment Higher effects for older and long-term unemployed Transitional employment in non-profit projects: Causal evaluation strategy: control group design (matching) positive effects on employment (after lock-in) Higher effects for women, long-term unemployed, unemployed with health issues 3
Evidence from counterfactual impact analysis of LMP in Austria Instruments handled by PES-officers Placement counseling Causal evaluation strategy: random controlled trial Result: shorter unemployment spells due to transition into employment as well as into inactivity Heterogeneity: more effective for the long-term unemployed, less for unemployed with health issues and the young Longer period of UB receipt Causal evaluation strategy: regression discontinuity design on age limits (52 instead of 39 weeks of entitlement with age 50) No significant effects on transition into employment but less transition into inactivity Higher effect on transition into inactivity for women due to means-testing of Unemployment Assistance (following UB) More sanctions (withdrawal of UB) Causal evaluation strategy: regional variation in the frequency (strictness) of the application of sanctions No significant effects on transition into employment but more transition into 4 inactivity
Concerns about effectiveness of workforce development programs Large heterogeneity of treatment effects for different types of training measures Both, with respect to target groups and types of measures Low effectiveness in particular for the low skilled unemployed Unobserved heterogeneity of training measures A more differentiated view of the type, content and quality aspects of the training measures is needed to assess (possible reasons for) their effectiveness More informative data required Heterogeneity in regional strategies (regional PES branches) Training measures cover a large share of labor market policy expenditures High share of low skilled unemployed, while skill requirements increase Public debate about effectiveness of PES training measures => Training effectiveness strategy within the PES 5
Research agenda to support an ongoing effectiveness strategy Improving available information A more precise coding of PES training measures is required to obtain insight on the effects of certain types of training: Learning content of measures (e.g. occupational skills, language, social skills, …), costs and duration, target group, modular combination with other measures, structure of participants in courses, competition of courses, … Exploiting not encoded data records, combining available information Identification of causal impacts of training measures Effect heterogeneity with respect to types of training, target groups, … (Fiscal) efficiency Comparison of performance (impacts) between regions To take into account differences in regional composition of unemployed … … and the regional labor market situation Comparability is required to identify successful regional strategies 6
Impact analysis for individual types of measures Counterfactual analysis of treatment effects Control group design: Nearest Neighbor Propensity Score Matching Applied separately for a regional breakdown based on a suitable aggregation regional PES-branches Indicators for labor market outcomes Employment, unemployment, out-of-labor force measured after x months after entry into treatment, or as days within a period after entry into treatment, time-to-job duration, retention rate in employment, entry-wage, … Problem of comparability of outcomes and treatment effects between regions (or different periods) 7
Comparison of program performance between regions Regional outcomes of training programs may differ for several reasons program effectiveness participants characteristics labor market conditions Objective: Fair comparison of regional program effectiveness Personal characteristics and local labor market conditions affect labor market outcomes of (potential) participants in programs Raw (unadjusted) comparison would judge regional program performance unfairly … … and would set unintended incentives (cream-skimming) => Adjustment of regional outcomes/effects for composition of participants and labor market conditions is required 8
Comparison of program performance between regions Well-established method for adjusted performance measurement (see Eberts et al., 2018) Statistical estimation: how much of the variation in individual outcomes is due to individual characteristics and local economic conditions … … and how much is due to the program performance (“value added”) 𝒋𝒌 𝒋𝒌 𝒌 𝒋𝒌 �� … measured outcome/effect for participant i in region j 𝑍 𝑌 �� … is a vector of personal characteristics or labor market conditions for participant i in region j and B … is a vector of estimated coefficients for 𝑌 �� � … is the estimated performance of region j 𝑋 𝜻 �� … is an error term The regional program performance 𝒌 equals the average outcome/effect � measured in the region j , minus the outcome that would be expected given the average characteristics of participants and labor market conditions � in region j 𝒌 𝒌 𝒌 9
Comparison of program performance between regions Estimation on individual participant data makes full use of heterogeneity across individuals (to include composition effects adequately) Comprehensive information on participants (and other unemployed) available Regional control group design would also allow for comparison of (causal) effects between regions Average treatment effect for the participants Choice of estimation model OLS, logistic regression, spatial models 10
Data background Merged administrative individual datasets Administrative PES-data: Individual characteristics (date of birth, gender, level of education, existence of health restrictions, care responsibilities ...) Unemployment episodes on daily basis, including information on participation in active measures and other temporary reasons for interruption of UB receipt Interventions by PES (appointments, placement suggestions, assignment to training, sanctions, ...) Austrian social security records: matched employer-employee dataset regularly processed at WIFO Employment relationships (duration on daily basis, wages on yearly basis) Employee characteristics (age, gender, full labor market career back to 1972, ...) Employer characteristics (industry, number of employees, ...) 11
Potential problems Relevant personal characteristics or regional labor market conditions may not be fully observable Omitted variables may lead to based results Program performance may not be separable from other policies Training programs may interact with placement service of the PES Adequate specification of the programs considered is necessary for performance measurement Homogeneous measure vs. policy system? Correspondence between policy(-level) and performance indicator Is the outcome substantially influenced by the policy? 12
Recommend
More recommend
