DANTE Visits to NOCs - 2012 TF-NOC, Poznan 13 December 2012 - PowerPoint PPT Presentation

Serving European NREN’s DANTE Visits to NOCs - 2012 TF-NOC, Poznan 13 December 2012 Creating the global research village

Agenda Serving European NREN’s Purpose Scope Key areas of interest Completed Changes Future Plans 2 Creating the global research village 2 2

Why? Serving European NREN’s DANTE NOC 3 years old Challenge assumptions and avoid stagnation Funding under increasing pressure Find new efficiencies and cost savings Increase NOC’s usefulness Identify and offer new services 3 Creating the global research village 3 3

Clarification Serving European NREN’s Benchmarking “… used to measure performance using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of performance that is then compared to others.” Our work not so specific or quantitative More “Fact Finding” 4 Creating the global research village 4 4

Scope Serving European NREN’s NOCs Visited NORDUnet (Dec 2011) SWITCH (Jan 2012) PSNC (Jan 2012) Indiana University GNOC (Jul 2012) Selection Criteria R&E Networks International/continental footprint Optical transmission and IP network 5 Creating the global research village 5 5

Key Areas of Interest Serving European NREN’s NOC (Operations) Organization Out of Hours Activity Percentage of incidents Level of Coverage Processes and Standards Key Performance Indicators (KPIs) & Service Level Agreements (SLAs) Additional Services 6 Creating the global research village 6 6

NOC Organization Serving European NREN’s Numbers and types of operations staff Org Teams Office Hours DANTE Service Desk, SD: 24/7 NCC (8), NCC: 0600-2100 Mon-Fri Specialists (12) Specialists: “9 to 5” Indiana U NOC Techs (23), NOC: 24/7 Engineering (25) Engineering: “9 to 5” NORDUnet NOC (7), NOC: 0700-1900 Specialists (11) Specialists: “9 to 5” SWITCH Global LAN (9) “9 to 5” PSNC Operators (6) Operators: Overnight & weekends NOC (14) NOC: 0800-1700 Specialist (20) Specialists: “9 to 5” 7 Creating the global research village 7 7

Out of Hours Activity Serving European NREN’s Organization Out of Hours Cover Percentage of Critical Incidents Out of Hours DANTE SD & On call NCC 40% engineer Indiana Uni NOC & On call engineer 33% NORDUnet On call NOC engineer 4% SWITCH Monitoring service & Call [1 or 2 times per year] list PSNC NOC Operators & 10% Call list (engineers) 8 Creating the global research village 8 8

Processes and Standards Serving European NREN’s Organization Processes and Standards DANTE Specific Work Instructions for SD; Procedures and Guides for NCC. ITIL based. Indiana Uni All documentation kept in WebGUI wiki – both GNOC-wide and GNOC network specific (in separate areas) NORDUnet All documentation in Confluence. SWITCH Some documentation in Wiki; good use of annotation in configs; comprehensive fibre documentation PSNC No formal adoption of standards but NOC generally follow ITIL principles 9 Creating the global research village 9 9

KPIs and SLAs Serving European NREN’s Organization KPIs SLA DANTE Ticket open – 15 mins Multi-homed: 99.999 / 99.9% First update – 60/90 mins Single homed: ~99.4% Time to fix: 5 hours Indiana University [Ticket open <20 mins] Bespoke, based on information flow NORDUnet [Ticket open 15/30 mins] TBC SWITCH Considered but not Multi-homed: 99.99% (inc PM) implemented Single-homed: 99.8% (exc PM) PSNC Ticket queue length (but Leased fibres – MTTR 10 hrs not formal) 10 Creating the global research village 10 10

Additional Services Serving European NREN’s Organization Services DANTE NSHaRP; E2ECU; MDSD Indiana Uni GNOC “Software as a Service” NORDUnet Adobe Connect; WAYF; VM servers; IP peering SWITCH AAI, PKI, eduroam; VC; e-collaboration PSNC Eduroam; VC; DNS; TERENA certificates; archiving 11 Creating the global research village 11 11

Follow Up – Completed Changes Serving European NREN’s Planned work conducted after not before core working day 90 minute shift overlap Ad hoc briefings, training DANTE Operations team divided into specialisations IP, Transport and Security specialists NOC Tier 3 specialists moved to respective teams 12 Creating the global research village 12 12

Future Changes Serving European NREN’s Magdalene contract expires March 2014. Options Re-tender for fully outsourced, off-site Service Desk OOH off-site Service Desk & in-hours in-house Service Desk Fully in-house Service Desk In-house Ticket System Greater control Minimise risk and cost of selection Commercial Off the Shelf Focus on popular systems (JIRA, RT, OTRS) 13 Creating the global research village 13 13

Special Thanks to Serving European NREN’s NORDUnet Jorgen Qvist, Freisa Perez SWITCH Willi Huber, Felix Kugler, Ernst Heiri PSNC Szymon Trocha and all in PSNC NOC and Engineering Indiana University Global NOC Jim Williams, Marianne Chitwood, Brandon Beale, Luke Fowler, Jon Paul Herron, Stephen Peck, David Jent 14 Creating the global research village 14 14

Serving European NREN’s Questions / Discussion Creating the global research village