Rules of Conduct • Established to maintain the confidentiality of ALL award applicant information • Ensure the fairness of the examination process • Aligned with Baldrige standards for state awards • A pledge by the Board of Examiners to uphold professional principles • Pertains to the entire Board of Examiners, including Judges 35
Potential Conflicts • Employer – Past, present or future • Previous evaluation of applicant – Within last 3 years • Key suppliers, customers, or competitors – Those critical to your organization – Roughly 5% or greater of your business • Financial holdings 36
True or False No one can serve on the Board of Examiners if s/he has not attended the three-day training course. True False 37
True or False The team leader is the only examiner that may contact the applicant after the site visit to clarify responses as we complete the final report. True False 38
True or False To meet critical team deadlines, it is acceptable for a friend or colleague who is a trained examiner to assist me in editing my observations and/or feedback-ready comments. True False 39
True or False When doing the independent review, I should visit the applicant organization’s website and social media sites for additional information. True False 40
True or False My service on the 2019 Board of Examiners ends once the Panel of Judges meets and the awards are announced. True False 41
True or False Cell phones should not be brought to the applicant’s facilities during the site visit. True False 42
True or False At the conclusion of my site visit, TNCPE will automatically reimburse me for the appropriate site visit per diem. True False BONUS: 2019 site visit per diem is $________. 43
True or False The conference and awards banquet is the proper time to approach and congratulate the organization I reviewed. True False 44
True or False Only organizations headquartered in Tennessee may apply for TNCPE awards. True False 45
True or False The amount of time I should expect to spend evaluating my applicant, including site visit, is 45-50 hours. True False 46
Multiple Choice • If upon arrival at the site visit, an Examiner suspects s/he may have a conflict of interest, s/he should: a) ask the applicant contact person if s/he objects to their participation b) refrain from participating in site visit questioning and turn focus to closing meeting c) immediately call the TNCPE office 47
Multiple Choice If your assigned applicant approaches you with an offer of employment: a) you may accept only if it has been 3 or more years since your site visit b) you may accept only if it has been 5 or more years since your site visit c) you may accept, provided the applicant sends a signed waiver to the TNCPE office d) you may not accept 48
Fill in the Blank If a second-year examiner is unable to fulfill his/her team obligations, TNCPE will send an invoice for the examiner training fee in the amount of $___________. 49
When In Doubt… KEEP CALM AND CALL TNCPE
Your Commitment • As a member of the TNCPE Board of Examiners, will you uphold the Rules of Conduct and Code of Ethical Standards? • Please sign and return the Code of Conduct • Welcome to the 2019 Board of Examiners! 51
Independent Review | Consensus | Site Visit PUTTING IT ALL TOGETHER
How Am I related to Baldrige/TNCPE? 53
Applicant Assessment Process Self a ssess Address Apply OFIs Receive Examiner 2 Feedback review & Report site visit 1 3 Award 5 4 54
Examiner Evaluation Cycle Complete June & July Pre-work & Training Finalize Independent August Feedback Review Report Site Visit Consensus Review Review 55
IR Review Review for Conflict Click “Done”! • Criteria • Key Factors • Analyze: PROCESS Prepare Log on to the system Consensus items (ADLI) 12 process • 6-10 Observations items • Score Participate in Consensus meeting Review the Framework • Criteria Prepare Site Visit • Key Factors issues Read the Application • Analyze: RESULTS (LeTCI) 5 results Complete Site Visit • 6-10 Observations items Create Key Factor List • Score Prepare Final Scorebook 56
Review – Identify Key Factors • What is a “key factor”? • Where do I find key factors? • How and where do I record key factors? • How many should I have? A key factor is an attribute of an organization or its environment that influences the way the organization operates and the key challenges it faces. Examiners will use key factors to focus their assessments on what is important to the applicant. 57
58
Item Evaluation Steps • Criteria – Read the Criteria item questions • Key Factors – Select 4-6 most relevant key factors from master list • Analyze the Application – Read relevant section of the application to identify processes and gaps ; analyze using process evaluation factors (A/D/L/I) • 6-10 Observations – Select and outline 6-10 strengths and OFIs 12 Process Items • Score the item • Criteria – Read the Criteria item questions • Key Factors – Select 4-6 most relevant key factors from master list • Analyze the Application – Read relevant section of the application; group results and identify gaps; analyze using results evaluation factors (Le/T/C/I) • 6-10 Observations – Select and outline 6-10 strengths and OFIs 5 Results Items • Score the item 59
Review – Insightful Process Observations • Identify gaps between the Criteria questions and the applicant’s response. Are these gaps important to the applicant? • Look for key factors not addressed. Did the applicant… – Address all customer and stakeholder requirements? – Consider all employee groups and segments? – Set measures for all relative goals/objectives? – Address all strategic challenges? 60
Review – Insightful Results Observations • Identify gaps between the processes described in Categories 1-6 and the results in Category 7. Are these gaps important to the applicant? • Use the applicant results tracker • Group results charts to provide new information by… – Common benchmark or comparison – Area to address – Integration with process item 61
Assignment Examiner Item 1 7.4 2 7.5 3 2.1 4 4.1 5 5.2 6 6.2 62
Exercise – Independent Review • Review the Criteria questions for your assigned item • Using the process item worksheet or the results item worksheet : (Process Item: pg. 199-203; Results Item: pg. 205- 209) – Select 4-6 relevant key factors; consider what you are expecting to see (5-10 min.) – Read LifeBridge’s response (15-20 min.) – Analyze (30 min.) • Note key processes [OR groups of key results] and any gaps • Assess processes/results using A-D-L-I or Le-T-C-I • Consider B/O/M • Outline 6-10 strength and OFI observations – Score your assigned item (5 min.) – Do NOT write feedback-ready comments yet 63
64
Examiner Evaluation Cycle Complete June & July Pre-work & Training Finalize Independent August Feedback Review Report September Site Visit Consensus Review Review 65
Consensus Review Review for Conflict Click “Done”! • Criteria • Key Factors • Analyze: PROCESS Prepare Log on to the system Consensus items (ADLI) 12 process • 6-10 Observations items • Score Participate in Consensus meeting Review the Framework • Criteria Prepare Site Visit • Key Factors issues Read the Application • Analyze: RESULTS (LeTCI) 5 results Complete Site Visit • 6-10 Observations items Create Key Factor List • Score Prepare Final Scorebook 66
67
68
69
70
71
Preparing an Item for Consensus Meeting 1. Review the Criteria questions 2. Assign 4-6 key factors 3. Read applicant’s response 4. Review team members’ Independent Reviews SYNTHESIZE WRITE SCORE REVISE PREPARE Synthesize team Draft 6-10 feedback Score the item based Revise comments Prepare to present members’ IR into comments based on on your consensus based on feedback your items during the 6-10 comment topics your chosen topics review from your backup team’s consensus meeting Develop rationale: Use TEST Provide feedback to B/O/M, evaluation your partner factors, and initials 74
Remember… • It is your job to select the MOST insightful 6– 10 pieces of information • Not all Criteria questions are equally important to every applicant! • Tell the applicant something it doesn’t already know… 75
Summary of Consensus Changes • Use the applicant’s name (LOTS) in your comments • Put strength and OFI comments in order of importance! • List initials AND B/O/M in your rationale • TEST can be spelled – STE – ETS – Or any other variation!
Exercise – Consensus Part 1 • Move to a table with others who completed the same item • Using your independent reviews, work as a team to select 4-6 key factors for your item • As a team, select 6-10 comment topics (strengths and OFIs) • What is your rationale for each comment? (initials, B/O/M, evaluation factors) • As a team, score your assigned item 77
Table Assignment Examiner Item 1 7.4 2 7.5 3 2.1 4 4.1 5 5.2 6 6.2 78
Exercise – Consensus Part 2 • Select one OFI comment topic from your team’s list • Independently write your assigned feedback-ready comment on the worksheet provided (pg. 211-212) • Return to your original table • Get feedback on your comment from someone at your table (your backup) • Revise your comment based on the feedback 79
Comment Writing Tips Is the comment Criteria-based? Is it insightful? T = Start with the “topic sentence” or main point E = Give 1 – 2 supporting examples from the application S = Explain the “so what” (relevance) to the applicant by linking the comment to a key factor T = Try it out loud! 80
Examiner Evaluation Cycle Complete June & July Pre-work & Training Finalize Independent August Feedback Review Report September September & Early Site Visit Consensus October Review Review 81
Site Visit Review Review for Conflict Click “Done”! • Criteria • Key Factors • Analyze: PROCESS Prepare Log on to the system Consensus items (ADLI) 12 process • 6-10 Observations items • Score Participate in Consensus meeting Review the Framework • Criteria Prepare Site Visit • Key Factors issues Read the Application • Analyze: RESULTS (LeTCI) 5 results Complete Site Visit • 6-10 Observations items Create Key Factor List • Score Prepare Final Scorebook 82
2018 Site Visit Strengths . . . the team was very thorough and worked diligently to understand our complex organization. They were professional but very approachable. . . . The visit helped to re-establish some of the gaps that we recognized during the application phase, and we look forward to the feedback report to see other areas that we need to address as an organization. 83
2018 Site Visit OFIs . . . We received the site visit schedule and several pages of requested documents and broadly worded topics just a week prior to the visit. That made full preparation difficult. . . . One examiner stated his questions too harshly to frontline staff. He was a bit condescending with his approach to several of the staff involved with the frontline staff focus group that the examiners requested. We had to ask the team lead for the questions that the examiners had established for the site visit, and they were received two days prior to the site visit. …With being a government agency, some of the thought process and questions can be very difficult to answer or connect to. It can be like fitting a square peg in a round hole. …Sometimes the team asked questions about things outside our control that we can't impact and added no value for us. 84
Site Visit Video 85
Site Visit Steps • Complete SVI form in Scorebook Navigator first – Documents, interviewees – Site Visit Issue only (verify/clarify questions) – The team leader sends these lists to the applicant 10-14 days before site visit • Then complete a SVI worksheet in Word for each issue • The applicant will receive your site visit issues (verify/clarify statements) and requested documents but not your prepared A-D-L-I questions or strategy 86
Exercise – Site Visit • Write a site visit issue based on the comment you wrote in the last exercise (verify/clarify) • Complete the Scorebook Navigator form (pg. 214) • Complete a Site Visit Issue Worksheet with questions you will ask the applicant (pg. 213) – A-D-L-I or – Le-T-C-I 87
Site Visit Findings • Use your SVI Worksheet (Word) to take notes during site visit • Team leader will collect these when complete (and save in case there are questions) • Document your evidence/findings in the Scorebook Navigator form with a conclusion: – OFI remains – Delete OFI – OFI becomes a strength • Update your comment to match the findings 88
“The secret to success is constancy of purpose.” Benjamin Disraeli FINALIZING AWARD RECOMMENDATIONS 89
Key Themes Score Summary Award Recommendation 90
Key Themes • Review key themes before you leave the post- site visit meeting • Do they appropriately reflect the applicant’s maturity? • Do they tell the applicant what critical few things it must do to move to the next level? • Are they traceable to the item comments ? 91
Scoring Your Applicant • Review item scores and complete the Score Summary Worksheet – Are item scores supported by the item comments? – Are scores stratified over all of the items? • Do not change scores in Scorebook Navigator! – Indicate on worksheet if score would move up or down by scoring range • Select a scoring band for the applicant – Does the scoring band reflect the applicant’s maturity? – Does the scoring band align with the key themes? 92
Score Summary Worksheet • Developed by Panel of Judges • Completed by team on post-site visit day • Captures changes in score by item from consensus to site visit • Conditional formatting to help show where new scores may land and tie to scoring bands • Covered in team leader webinar and JIT webinar 93
SCORNG SUMMARY FOR APPLICANT # 1709 Item Number Consensus Site Visit Score Strengths (+) Strengths (++) OFIs (-) OFIs (- - ) Scoring Range Strengths (+) Strengths (++) OFIs (-) OFIs (- -) Comments 7 7 KT a/b 5 3 KT c/d 1.1 55% 39 3 2 4 0 50% 35 65% 46 3 2 3 0 Deleted 1 OFI 1 strength merged with 1.2 45% 23 4 0 3 1 30% 15 45% 23 3 0 2 1 another strength; deleted 1 OFI Moved 1 strength from 4.1 to 2.1; changed 2 OFIs to 2.1 45% 20 4 0 4 0 50% 23 65% 29 6 1 1 1 strengths; changed 1 OFI to (- -) Changed 1 strength to ++; 2.2 50% 20 3 0 3 0 50% 20 65% 26 2 1 2 1 changed 1 OFI to (- -). Added 2 strengths; deleted 1 3.1 30% 12 3 0 1 2 50% 20 65% 26 5 0 3 0 OFI; added 1 OFI; no more (- -) OFIs 3.2 35% 16 3 0 2 1 50% 23 65% 29 3 0 2 1 Moved 1 (++) strength to 2.1; deleted 1 strength; delete 1 (- 4.1 40% 18 3 2 3 1 30% 14 45% 20 2 1 2 0 ) OFI and 1 (- - ) OFI. Moved 1 strength to 5.2; 4.2 60% 27 4 2 2 1 50% 23 65% 29 3 2 1 1 deleted 1 OFI Deleted 3 OFIs; added 1 OFI 5.1 65% 26 4 0 3 0 70% 28 85% 34 4 0 1 0 Moved 1 strength from 4.2 5.2 60% 27 5 1 2 0 50% 23 65% 29 5 2 2 0 and made it a (++); deleted 1 OFI; added 1 OFI Deleted 3 OFIs;changed 1 6.1 40% 18 3 0 5 0 30% 14 45% 20 3 0 1 1 OFI to a (- - ) 6.2 45% 18 4 0 4 0 50% 20 65% 26 4 0 2 0 Deleted 2 OFIs 263 255 338 Deleted 1 OFI and added 1 7.1 35% 42 3 0 2 1 30% 36 45% 54 3 0 2 1 OFI 7.2 25% 21 2 0 4 0 10% 9 25% 21 2 0 3 1 Changed 1 OFI to (- - ) 7.3 40% 34 3 0 3 0 30% 26 45% 38 3 0 2 0 Deleted 1 OFI 7.4 20% 16 2 0 2 2 30% 24 45% 36 2 0 2 2 Deleted 2 OFIs and added 1 7.5 40% 32 3 0 2 0 50% 40 65% 52 3 0 1 0 OFI 145 134 202 408 389 539 Scoring Bands (Calculated) Approach 4 3 5 Results 2 2 3 Scoring Bands (Team Recommendation) Approach 5 Results 3
Table Exercise • Review the Room Scoring Summary as a class • What process and results band is LOTS in after consensus? • As a table, review the scoring band descriptors and determine which most accurately describes LOTS after site visit - Process - Results
TNCPE Recognition Levels Level 4 Level 3 Level 2 Continue to improve and build Commitment and practice Beginning to commit and Principles upon outstanding results and implement excellent systems Exhibit processes that serve as Demonstrate significant Put in place some key Processes role models for others, with progress in building process improvements some at or near “best-in- systematic processes class.” Multiple questions Overall questions Basic questions Effectively Address Demonstrate management Results for some key Process improvements are Results excellence with superior processes show directly attributable to a results over time, directly improvement that is fact-based improvement attributable to a systematic directly attributable to a process. Results may or well-deployed improvement systematic approach. may not be evident; approach. however, a measurement system should be in place. Outstanding examples of high- Many practices from which Practices performance organizations other organizations can learn and grow. 96
Level 1 organizations are . . . . . . interested in adopting and applying performance improvement principles 97
98
Level 2 Award Recommendation ABC is in the early stages of developing systematic approaches to address the basic questions of the process categories, as evidenced by: listening to customers through XX system; conducting an annual Strategic Planning retreat; recruiting and hiring of new employees; emergency preparedness and prevention planning; and the existence of processes that promote ethical standards of conduct. While there are multiple processes in place, many are not fully deployed and/or lack the means to establish cycles of learning that could lead to continuous improvements for the organization. ABC collects large amounts of data but is still in the early stages of determining how to effectively analyze this data for use in strategic planning. Segmentation of data and important comparison results that would allow the organization to benchmark itself against competitors in a rapidly changing healthcare environment are missing. 99
Level 3 Award Recommendation XYZ has demonstrated significant progress in building systematic processes that effectively address the overall questions of each item . . . It has established documented processes for most of the identified key work systems. These processes are deployed and aligned throughout the organization appropriately to meet the key business needs of the organization. Some of the processes remain informal as it is still a small company at this point. Organizational learning has mostly been the result of the new owner and is now in the early stages of being transitioned to the entire workforce. 100
Recommend
More recommend
