“I get NOC'd down...” ...but I get up again T owards excellent NOC customer service even when things go wrong Gareth Eason (HEAnet) TF-NOC Meeting, Barcelona, 2 nd Sep 2010
Who am I? • Gareth Eason Network Operations Manager, HEAnet John Boland (CEO) Gareth Eason (Network Operations Manager) Brian Nisbet (NOC Manager) NetOps team NOC (Virtual T eam)
Remember Copenhagen? This is only presentation #1 (of at least 2 ) ... because this one has no answers, only questions...
Barcelona This is presentation #2 (of at least 2 ) ... where we realise there are no answers but there are some good ideas ...
Overview I want to change: 1. how you think about your NOC 2. how your NOC thinks about its customers
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
TF-NOC: Work Items ● NOC Users ● Who are they? ● Agreements ● What agreements should exist? ● What should they contain? ● User Relations ● How do they relate to a NOC? ● Escalation & Priority ● When? How? How do we ensure resolution?
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
“Expect success, but plan for failure” -- Unknown
Who you gonna call? • NOC only gets called when a failure occurs. • Be proactive – avoid failures! Resilient solutions Monitoring and alerting Alerting relevant to client • Consider alerting direct to the client The NOC calls the client on detecting a problem Web portal displaying monitoring for client
Who you gonna call? • Make sure NOC is available Designated call handler Designated escalation paths Consider a recorded message saying there is a problem • Make sure problem is resolved Designated call handler Designated problem handler • Make sure customer stays informed!
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
Failure of a supplier • Measure how do we define (there is) a problem? • Inform Make sure the supplier knows Make sure the rest of your organisation knows • Plan to resolve Agree what you want (from supplier) • Agree timeline and metrics Agree how you will both know it's resolved
Crisis Meeting • Second last resort • Collect data and metrics before the meeting • Invite account manager • Invite account manager's manager • Avoid covering old ground • Avoid assigning blame • ...
Crisis Meeting 1. Agree there is a problem 2. Agree the scope of the problem 3. Agree a resolution to the problem ● Might be a resolution ● Might be periodic updates if longrunning problem ● Define what you want to see from supplier 4. Agree how you'll know the problem is resolved 5. Agree the next meeting date
Next steps • Supplier failure: Escalate further (if useful) Contractual penalties Use alternate suppliers, if any • Last resort: Material breach of contract Material breach of contract / cessation of contract ...might mean procurement again. Dependant on legal jurisdiction
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
Client Problem 1. Understand their problem 2. Triage 3. Define action plan (with client) 4. Implement Plan 5. Report on success (or otherwise) of plan 6. GOTO step 1.
Understand their problem • Often biggest hurdle • Understand direct and indirect implications Loss of revenue? Loss of face? Media? Press? Publicity? • Knowing your client well will help You do have regular meetings with your clients, right? • Find out what the client is actually trying to do!
Triage • Can we solve the whole problem? • Busy website Are some parts of it busy? Can we isolate them? Can we replicate the service • Loss of connectivity Are there vital services for which a few 3G modems will do? • Figure out what the customer actually needs
Action plan • 3 hour (emergency) Emergency Reactive • 24 hour • 3day • 7day Proactive • 1 month • 3 month • Long term (1 year, etc.) Strategic
Implement Plan • Put everything in place. • Monitor towards success criteria
Report • Discuss measurements with client • Did things work as expected? • Were the success criteria met? • Were they the right success criteria? • Go back to step 1 : Understand their problem
Business As Usual • Agree (with customer) criteria for returning to Business As Usual • Learn from failure Make the business as usual for NOC better business!
Overview • TFNOC Work item • Planning for failure • When a supplier's service fails • When a client's service fails • Lessons learned
Understanding • Understand the problem Educate suppliers about your pain points Educate supplier on implications of problems (e.g. Delivering a router late means a client cannot register students at a new campus.) Learn from customers – what are their pain points?
Communication is key • Deal with the 'Mars' and 'Venus' compexes Allow NOC to listen to problems – some customers just want to let you know. (Recorded message!) Enable NOC to triage, work around and fix • Make sure communication doesn't slow down fault resolution Designated person on point for comms. Designated person on point for escalations.
Work around problems • Don't be afraid to work around problems Root cause repair might not be quick enough Understand the customer's view of the problem – maybe something less than a complete fix is good enough. • Agree and stick to action plans Agree plan with customers Ensure your NOC knows the plan. Ensure your boss knows the plan!
Don't fear change! If the action plan is not working... ... CHANGE IT!
Do one thing today to change how one customer thinks about your NOC in a positive way .
Recommend
More recommend