ROI: Estimates and Forecasts don't have to be that complicated! - PDF document

ROI: Estimates and Forecasts don't have to be that complicated! Sometimes you need a quick, un-complicated method for an ROI on a performance improvement or training intervention. While an in-depth ROI analysis - developed along-side the intervention - may be the ideal, it is not always feasible. In some circumstances a credible forecast or estimate is the best option for providing ROI. The focus of this session is a discussion of ideas and tips for identifying credible measurements for ROI. Return On Investment (ROI) Excerpts from the ROI Institute and Phillips latest book, Real World Training Evaluation (2016). A systematic approach to program evaluation - includes three key components: 1. A 5 level framework 2. A 10 step process model 3. The 12 guiding principles Five Level Evaluation Framework Levels of Key Questions Answered Evaluation  Level 0: Inputs How many people?  and Indicators How many hours?  How many offerings?  What are the costs per person?  Level 1: Was the program relevant to participants’ jobs?  Reaction and Was the program important to participants’ success?  Planned Action Did the program provide new information?  Do participants intend to use what they learned?  Would participants recommend it to others?  Is there room for improvement with facilitation, materials, and the environment?  Level 2: Do participants know what they are supposed to do with what they learned?  Learning Do participants know how to apply what they learned?  Do participants know next steps?  Are participants confident in their ability to apply what they learned?  Level 3: How effectively are participants applying what they learned?  Application How frequently are they applying what they learned?  How much of what they learn do they applying?  If they are applying what they learned, what is supporting them?  If they are not applying what they learned, why not?  Level 4: Impact So what?  To what extent does participant application improve measures of output, quality, cost, time, customer satisfaction, job satisfaction, work habits, or innovation?  How do we know it was the program that improved those measures?  Level 5: ROI Do the monetary benefits of the improvement in key measures outweigh the program costs? Program Benefits  BCR = Program Costs Net Program Benefits  X 100 ROI = Program Costs

Ten Step Process Model Source: ROI Institute 12 Guiding Principles 1. When conducting a higher-level evaluation, collect data at lower levels. 2. When planning a higher level evaluation, the previous level of evaluation does not have to be comprehensive. 3. When collecting and analyzing data, use only the most credible sources. 4. When analyzing data, select the most conservative alternatives for calculations. 5. Use at least one method to isolate the effects of the program. 6. If no improvement data are available for a population or from a specific source, assume that no improvement has occurred. 7. Adjust estimates of improvements for the potential error of the estimates. 8. Avoid use of extreme data items and unsupported claims when calculating ROI calculations. 9. Use only the first year of benefits in the ROI analysis of short-term programs. 10. Fully load all costs of the program when calculating ROI. 11. Include intangible benefits as measures that are purposely not converted to monetary values. 12. Communicate the results of the ROI study to all key stakeholders. Different approaches can still meet the standard  Most accurate – invest more to reduce risk of making a wrong decision  Shortest route – avoid time and cost commitments but get credible information

Phillips (2016) Examples of Business Measures AAMA Example Project  Sample Survey  Alignment of questions to measurement levels  Appendix H. Average Attendance Cost  Appendix I. Weighted Average of Wages  Estimated costs to attend conference  ROI Forecast of Conference Program  Benefit Cost Ratio and ROI

Five steps to calculating monetary benefits (Phillips, 2016) 1. Define the unit of measure. 2. Determine the value of the measure. 3. Determine the change in performance. 4. Annualize the change in performance. 5. Calculate annual monetary benefits. Where you do not have data to determine the value of a measure, you can get a good estimate of monetary value by asking credible sources three questions:  What happens?  What is it worth?  How confident are you? For example, a manufacturer start having an absenteeism problem but the company does not have a standard value or historical costs. An estimate using credible sources is much faster and cheaper than spending time and money to develop a standard value. In this example, supervisors are deemed a credible source since they are closest to the problem. Using a focus group of five supervisors, each supervisor was asked three questions. 1. What happens when one of your employees does not show up for work? 2. Given what happens when someone does not show up for work, how much do you think it costs you per day per absence? 3. How confident are you in your estimate? Supervisors gave several examples in response to question one: pay overtime to people who make up the work, work is unfinished, supervisor personally covers for absent employee. The estimates provided for questions two and three were used to calculate a monetary value: Q2 Responses Q3 Responses Adjusted Value 1. $1,500 x 0.95 = $1,425 2. $1,900 x 0.9 = $1,710 3. $2,300 x 0.8 = $1,840 4. $2,000 x 0.85 = $1,700 5. $1,850 x 0.9 = $1,665 Total $8,340 Average $1,668 This process works because:  The most credible sources provided the information.  Three specific questions were asked to ensure repeatability of the process.  Estimates were adjusted for error using a confidence level.  Result is the most conservative alternative.

ROI References and Resources Bashrum, M. (2012). Keep your training program funded: 10 steps to training ROI. The Public Manager. Retrieved from https://www.td.org/Publications/Magazines/The-Public- Manager/Archives/2012/Winter/Keep-Your-Training-Program-Funded Burkett, H. (2002). Evaluation: Was your HPI project worth the effort? In G. M. Piskurich, (Ed.). HPI essentials (pp. 153-175). Alexandria, VA: ASTD. Buzachero,V.V., Phillips, J., Phillips, P. P., & Phillips, Z. L. (2013). Measuring ROI in healthcare: Tools and techniques to measure the impact and ROI in healthcare improvement projects and programs. New York: McGraw Hill. DeTuncq, T. (2012). Demystifying measurement and evaluation. Infoline , no. 1211. Alexandria, VA: ASTD Press. Freer, K., Johnstal, S.P., & Minchella, K. (2014). Forecasting ROI in a National Conference Program: A Professional Healthcare Association. Unpublished report submitted to the ROI Institute. Phillips, P.P., & Burkett, H. (2001). Managing evaluation shortcuts. Infoline, no. 0111. Alexandria, VA: ASTD Press. Phillips, P.P., & Phillips, J.J. (2016). Real world training evaluation: navigating common constraints for exceptional results. Alexandria, VA: ATD Press. Phillips, J. J., Phillips, P. P., & The ROI Institute, Inc. (2008). The Measurement & Evaluation Series: Book 1. ROI fundamentals. San Francisco, CA: Pfeiffer. The value of learning: gauging the business impact of organizational learning programs. (2013). ASTD Research: Connecting research to performance , 5(4). Alexandria, VA: ASTD Press. Websites: http://www.roiinstitute.net/free-tools/ www.td.org http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html http://managementhelp.org/training/systematic/ROI-evaluating-training.htm