Presented by: Ken Phillips Phillips Associates February 21, 2018 Phillips Associates 1
Agenda 1. Discover meaning of term “scrap learning” & its impact on wasted organization resources & lost credibility with stakeholders 2. Analyze how to build an algorithm that predicts which learners are most & least likely to apply what they learned in a training program back on the job & which managers of the learners are likely to do a good and poor job of supporting the training 3. Examine the 3-phase, 9-step Predictive Learning Analytics methodology using data from an actual implementation
Scrap learning: What is it? Phillips Associates 3
Scrap Learning Term that describes the gap or difference between learning that is delivered and learning that is applied back on the job Coined by KnowledgeAdvisors, a CEB company (now Gartner) Phillips Associates 4
Phillips Associates 5
How big is the problem? Phillips Associates 6
Poll In the average organization, what percent of learning that is delivered ends up as scrap? A. 25% B. 45% C. 65% D. 85% Phillips Associates 7
Benchmark Study 1 45% Source: Confronting Scrap Learning CEB White Paper, 2014 Phillips Associates 8
Benchmark Study 2 > 15% Applied new skills back on the job Didn’t try to apply new skills back < 20% on the job Tried applying new skills back 65% on the job, but reverted back Source: Robert Brinkerhoff, 2004 Phillips Associates 9
View from individual organization level Phillips Associates 10
According to ATD 2017 “State of the Industry Report” Average per employee training = $1273 expenditure Average number of training = 34.1 hours consumed per employee Phillips Associates 11
Calculating Scrap Learning at individual organization level $573 $1273 X 45% = 15 34.1 hours X 45% = $1018 $1273 X 80% = 27 34.1 hours X 80% = Phillips Associates 12
View from individual program level (see formula on page 5 in handout) Phillips Associates 13
Houston, we have a problem! Source: James Lovell, Apollo 13 flight Phillips Associates 14
Predictive Learning The solution: Analytics ™ Phillips Associates 15
Definition Predictive Learning Analytics: Methodology for peering into the future, at the conclusion of a learning program, and predicting learner outcomes and actions, with the intent of changing those outcomes and actions for the better Phillips Associates 16
PLA Mission To provide L&D professionals with a systematic, credible and repeatable process for measuring and managing scrap learning using data driven decision making Phillips Associates 17
PLA vs. Traditional Learning M&E Predictive Learning Analytics Traditional Learning M&E Focuses on individual Focuses on programs or learners cohorts Predicts future likelihood of Describes what has certain behaviors and actions happened Phillips Associates 18
The PLA Methodology Peering into the future & predicting learner outcomes & actions Changing those outcomes & actions for the better Reporting your results
Phase 1: Step 1 Phillips Associates 20
Select a Learning Program Three Guidelines: Planned learning initiative not informal 1. learning event 2. Has a high profile Large number of participants are scheduled to 3. attend (40-60 for initial Calibration Cohort) Phillips Associates 21
Phase 1: Step 2 Phillips Associates 22
All 12 factors are aligned with what research has found to be the 3 components of training transfer Research sources: Baldwin & Ford 1988; Colquitt et. al. 2000; Scaduto et. al. 2008 Phillips Associates 24
3 Training Transfer Components Learning Learner Direct Influence Program Success! Attributes Control Design Learning is applied on the job Learner Work Environment Influence
Instructions 1. Form a group of 3, 4 or 5 persons 2. Keeping in mind the 3 training transfer components, brainstorm a list of factors known to contribute to training transfer (page 1 in your handout) Example : Training transfer increases when learners have an immediate opportunity to apply what they learned in a program back on the job (Work environment ) 3. Be prepared to share your ideas with the whole group
3 Training Transfer Components Learning Learner Program Success! Attributes Design Learning is applied on the job Learner Work Environment Phillips Associates 27
Program Design Factors 1. New information is acquired 2. Program viewed as relevant to learner & his/her job 3. Program viewed as important investment in one’s career development 4. Learner sees likely improvement in key department business metric if new information learned is applied 5. Learner is likely to recommend program to work colleagues Continued Continued Phillips Associates 28
Learner Attribute Factors 6. Learner is personally motivated to apply what was learned 7. Learner is confident in his/her ability to apply what was learned 8. Learner takes time to reflect on key lessons learned & how they can help improve performance 9. Learner views program as an opportunity to learn challenging new things Continued Continued Phillips Associates 29
Work Environment Factors 10. Managers actively engage learners, post-program, regarding what was learned 11. Work colleagues support learners, post-program, when applying new things learned 12. Learners have immediate opportunity to apply what was learned Phillips Associates 30
Create a Survey Convert 12 factors into survey items that First reflect content of target program Incorporate survey items into an existing Then Level 1 evaluation or administer as a separate survey Phillips Associates 31
Sample Survey Items How relevant is the (insert program name) program to you and your job? Extremely Not at all Relevant Relevant 7 1 2 3 4 5 6 How confident are you in your ability to apply the knowledge, skills and behaviors you learned in the (insert program name) program back-on-the-job? Extremely Not at all Confident Confident 5 6 7 3 4 1 2 Phillips Associates 32
Phase 1: Step 3 Phillips Associates 33
Case Study Company: Medical insurance company Improve operational Business objective: efficiency Continuous Process Learning program: Improvement Calibration cohort: 74 participants Phillips Associates 34
LAI Scores Most Likely to Apply At Risk of Not Applying Least Likely to Apply Phillips Associates 35
MTSI Scores Average score on factor measuring how likely manager is to be actively engaged Average LAI score for all employees reporting to same manager MTSI = difference between with learner post-program regarding what was learned & is an indication of manager support for training & is an indication of Mgr. Average & training transfer potential LAI Average Phillips Associates 36
Training Transfer Component Scores No statistically significant difference Statistically significant differences Lowest factor score
Phase 1: Step 4 Phillips Associates 38
Calculate Scrap Learning & ID Obstacles 30 days post-program collect data from random sample of Calibration Cohort participants using either a survey or focus groups and ask 3 questions : 1. % of program material applied back on job? 2. Confidence level of estimate? 3. Obstacles inhibiting application back on job? Phillips Associates 39
Scrap Learning Calculation
Obstacles to Training Transfer Management (11) Policies and Procedures (10) Communication (9) Personal (7) (6) Lack of time or resources Technology (4) Teamwork (4) Change (3) Phillips Associates 41
Phase 2: Step 5 Phillips Associates 42
Step 5: Where the Rubber Meets the Road Use data driven decision making to: 1. Identify solutions to mitigate or eliminate obstacles to training transfer 2. Target at risk & least likely to apply learners for reinforcement activities 3. Target managers with low or negative MTSI scores for help & support 4. Manage scrap learning baseline percentage
Phase 2: Step 6 Phillips Associates 44
Prediction without validation is nothing more than educated guessing at best and malfeasance at worst. Source: Ken Phillips Phillips Associates 45
Phase 2: Step 7 Phillips Associates 46
Recalculate Scrap Learning Using a new group of learners, collect data from random sample of participants using either a survey or focus groups and ask same 3 questions: 1. % of program material applied back on job? 2. Confidence level of estimate? 3. Obstacles inhibiting application back on job? Phillips Associates 47
Phase 3: Step 8 Phillips Associates 48
Phase 3 Step 9
Summary The issue of scrap learning has been around forever. But, what’s different today is that with Predictive Learning Analytics™ there now is a way to measure and manage it. Source: Ken Phillips Phillips Associates 50
Phillips Associates 51
Learn more about Predictive Learning Analytics Request our FREE ebook: The L&D Revolution: New Rules. New Tools. Phillips Associates 52
Recommend
More recommend
