Delaware Cost Study Progress Report Beyond Average Benchmarking - Use of the Delaware Cost Study data and Data Envelopment Analysis to Target Productivity Improvement and Promote Excellence in Resource Utilization Tom Eleuterio tommyu@udel.edu Manager, Higher Education Consortia Ti Yan yant@udel.edu Research Analyst, Higher Education Consortia Office of Institutional Research and Effectiveness University of Delaware
Big Picture Key decisions are often made at the department level to allocate faculty and financial resources to instruction, research and service. Benchmarking is used to provide straightforward information for program evaluation and strategic planning. How do you know if a program is efficient?
Outline Why We Need a Problem Space: A Method that Makes Adjusted Models for Better Benchmarking Benchmark w/ whom the Best Use of Available Different Types of Approach? on what? Data Resources Programs Data Envelopment Analysis (DEA) Basics and Application to Program-level Instructional Productivity and Cost Data Data Selection for Use in DEA Data Availability and Model Fit DEA Results and Discussion: Case Study Alternative DEA Models Case Studies to Match Specific Program Levels or Data Sparse Sample Spaces
Audience Poll How does your institution evaluate each program’s instructional costs and productivity?
Problem Space 1: Benchmarking with Whom Individual academic programs might not be aligned with their institutional classification. A program that only offers bachelor degrees is operating within a Carnegie class research high university. Aligning benchmarks for the departmental/program-level cost of instruction requires comparison across program-level peers. A program’s peers are not equivalent to institutional peers.
Problem Space 2 : Benchmarking on What All Institutional Costs Costs Allocated to Central Costs Departments Direct Direct Public Direct Externally Funded Academic Instructional Service Costs Research Costs Indirect Costs Research & Dept. Administration Costs (sep. budgeted) (sep. budgeted) Match Institutions submit their data and access our national Personnel : benchmarking norms for key performance indicators (KPI) such as: Faculty and Non-Personnel support staff Refined Means of Cost $ per Student Credit Hour by CIP2 or CIP4 Salary Refined Means of Cost $ per FTE student by CIP2 or CIP4 Three Benchmarking norms are produced: Carnegie Classes, Benefits Highest Degrees and percentage of Undergraduate Degrees.
Problem Space 2 : Benchmarking on What The averaged Cost per SCH or per Student does not reflect your relevant position as compared to available norms. Benchmarking should inform decision makers with specific, quantitative and measurable goals for continuous improvement.
To Address the Two Problems 1. Data-driven Comparator Group Selection by discipline finds out which institutions offer programs that are the most comparable in terms of instructional productivity. 2. Data Envelopment Analysis (DEA) identifies the best-performing units among the group members and examines the differences in output metrics. • Among those comparator groups, who produces the optimal combination of instructional outputs within the constraints of their resourced inputs? • Who is doing the best and how is this accomplished?
DEA (Data Envelopment Analysis) Extreme Relative Point Efficiency Efficient Frontier Method “Best" Virtual Producer (LP) Efficiencies: Efficiency Number Recommendations to obtain improved efficiencies
DEA - It’s all about evaluating EFFICIENCY Ways to be efficient Maximizing output with keeping input constant. Minimizing input with keeping output constant. Or both, at the same time. How to measure efficiency Efficiency scores – a score produced by the DEA model ranging from 0 to 1, with 1 being the most efficient. Total Weighted Output / Total Weight Input
Data Envelopment Analysis (DEA) DMU Input Output1 Output2 A 100 40 0 B 100 20 5 C 100 10 20 Decision Making Units (DMU) represent a group of organizational units that are compared in the process of DEA based on their measurements on a certain set of inputs and outputs. Output2 A,C: Efficient; B: Inefficient 25 • C E: “Best” Virtual DMU of B 20 • 15 10 E (29.2,7.3) 5 B Output 1 A 0 0 5 10 15 20 25 30 35 40 45
DEA: Illustration Fig 1 shows a set of DMUs with each consuming the same amount of a single resource (e.g. a fixed number of faculty members) and producing different amounts of two outputs: y1 (e.g. Lower Division Student Credit Hours) and y2 (e.g. Upper Division Student Credit Hours). Solid lines construct the efficiency frontier on which all DMUs are efficient ( efficiency # =1). Output 2 Two possible best-practice virtual producers for P 5 • are labeled as P’ 5 and P’’ 5 . P’ 5 can be achieved if y1 and y2 could be increased • proportionally as depicted in last slide. If y2 could not be increased for P 5 , the alternative • is to increase y1 solely to reach the efficiency frontier, shown as P’’ 5 . In the case of P6, only one possible best virtual • producer is labeled as P’ 6 . Output 1 Figure retrieved 5/25/2017 from http://deazone.com/en/resources/tutorial/graphical-representation
Assumptions when Applying DEA to Delaware Cost Study Results Tenure-tracked FTE are quite different from other types of FTE. No difference between SCHs taught by different types of faculty members. SCHs taught at different student levels are not replaceable. The ability to gain separate budget indicates the research and service ability . Define the Inputs and Outputs Input variables Output variables Total SCHs taught at different levels Number of FTE Instructional Faculty -Tenured/Tenure-eligible and All others - Lower/Upper Division, Graduate, Individual Instructions Total direct expenditures for instruction
Bef efore D DEA: d data-infor ormed p peer eer s select ection u using Englis lish C CIP 23.0 3.0101 f from f four c con onsecutiv tive y yea ears of of cos ost s t stu tudy d data a as case s e stu tudy on one • Latent Class Analysis (LCA), a subset of Structural Equation Modeling (SEM), was used to identify four comparator groups using a longitudinal dataset obtained from 2012 – 2015; sample n = 71. • Naïve Bayesian method was used to classify all one-year (2015) participants to the settled groups by LCA; sample n = 174. • DEA results and discussion: English CIP 23.0101 cluster #3 case study • Discussion of next steps possible for use by all Carnegie universities, colleges or departments
Data Envelopment Latent Class Analysis Naïve Bayes Clusters Analysis • CIP-specific • CIP-specific • CIP-specific • Four-year participants 4 • 2015 participants by FICE • Each individual group consecutive years ( 2012 thru peers from 2015 2015) • Missing data allowed from participants. earlier years ( 2012 thru • No missing data 2014). • Inputs: faculty • 6 metrics number and rank, • Highest degree (categorical) • Identifying refined groups of direct instructional $. • # of all degrees peers in one-year • % of bachelors in all degrees participation. • Outputs: SCHs at UG • % of UG OCS in total OCS and GR levels. • Group peers are comparable • % of TT OCS in total OCS in terms of the 6 metrics and • Efficiency numbers • Standardized research and public service $. ready for efficiency and Best-practice • Initial sample of 71 cases assessment in the next step. virtual DMU clustered in 4 groups. generated.
A Map of 174 participants in a CIP 23.0101 English, for The Year of 2015 Delaware Cost Study, classified by four groups.
31 Cluster #1 Institutions from the 2015 Delaware Cost Study with 174 total participants in CIP 23.0101 univname class hideg alldeg state univname class hideg alldeg state Boise State University R3 2BM ID University of Delaware R1 1BMD DE DePaul University R3 2BM IL University of Kansas R1 1BMD KS Florida International University R1 2BM FL University of Massachusetts Amherst R1 1BMD MA Georgia State University R1 1BMD GA University of Missouri - Columbia R1 1BMD MO Grand Valley State University M1 2BM MI University of Missouri - St. Louis R2 2BM MO Kansas State University R1 2BM KS University of New Hampshire Main Campus R2 1BMD NH Miami University - Oxford R2 1BMD OH University of North Carolina at Chapel Hill R1 1BMD NC North Carolina State University at Raleigh R1 2BM NC University of Tennessee - Knoxville R1 1BMD TN Northern Arizona University R2 1BMD AZ University of Utah R1 1BMD UT Northern Illinois University R2 1BMD IL University of Vermont R2 2BM VT Ohio State University - Main Campus R1 1BMD OH University of Virginia - Charlottesville R1 1BMD VA Simon Fraser University R2 1BMD BC Virginia Polytechnic Institute & State University R1 1BMD VA SUNY at Albany R1 1BMD NY Wilfrid Laurier University M1 1BMD ONT University of California - Irvine R1 1BMD CA Wright State University - Main Campus R3 2BM OH University of Central Florida R1 1BMD FL Youngstown State University M1 2BM OH University of Connecticut R1 1BMD CT
Recommend
More recommend
