Communication Systems Network Operations at CERN Gyorgy Balazs - PowerPoint PPT Presentation

Communication Systems Network Operations at CERN Gyorgy Balazs Terena TF-NOC Ljubljana Ljubljana CERN IT Department 15.02.2011 CH-1211 Genève 23 Switzerland www.cern.ch/ i t

Network Infrastructure Multi-purpose, multi-vendor network infrastructure Internet connectivity, Experimental instruments, LCG, Internet Exchange (CIXP) …Where the web was born 3 distinct multi-ten-gigabit 3 di ti t lti t i bit backbones 150+ very high performance routers 3’700+ subnets 2200+ switches (increasing) 2200+ switches (increasing) 50’000 active user devices 80’000 sockets, 5’000 km UTP cable 400+ starpoints 5’000 km of fibers (CERN owned) 5 000 km of fibers (CERN owned) 150 Gbps of WAN connectivity Extremely Dynamic environment → 2x more visitors than staff 1500 connection and change request / month 1500 i d h / h CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 2/11 www.cern.ch/ i t

Network management All centrally managed infrastructure 10+ years of development of our own NMS  ~5 software developers  Extremely high level of automation  500’000+ lines of code/150+ DB tables This allows for very limited manpower dedicated to operations CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 3/11 www.cern.ch/ i t

NOC Structure CERN Users On site On-site Service Desk Outsourcing Support Only IT Helpdesk Switchboard (Network) (Network) (Telephony) (Telephony) Network Operations Technicians All All Network Operations Network Engineer teams External Entities (CIXP, LHCOPN...) CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 4/11 www.cern.ch/ i t

NOC Structure: Staff Operations: 3 non-degreed specialists dedicated during office  hours Monitoring: – LAN, Telecom • Support: – LAN, Telecom • Engineer on duty: 6 engineers in daily rotation, ~10% load during E i d t 6 i i d il t ti 10% l d d i  office hours Monitoring: – LAN, WAN, Network services (DNS, DHCP...) ( ) • (Advanced) Support: – LAN, WAN, Network services, External entities • 1 engineer dedicated to CIXP and Hosting services  administration and support d i i t ti d t OWH by Computer Centre Operators  Monitoring, triggering urgent interventions – CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 5/11 www.cern.ch/ i t

NOC Structure: Tools Only commercial tool: CA Spectrum Cricket Nagios SMS PerfSONAR (Servers, External network) (Servers External network) (Routers, Servers) (Servers) (LHCOPN) + Homegrown Statistics and Sflow based network behaviour and anomaly detection tools CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 6/11 www.cern.ch/ i t

Front End Network Support Servicedesk → IT Helpdesk → Operations → Engineer teams Final stage of ITIL implementation Done on the organisation level, new ticketing tool g , g BMC Remedy Service-Now CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 7/11 www.cern.ch/ i t

Front End Network support tickets resolution ratio Week Total IT Helpdesk Operations Helpdesk (2011) Ratio 6 65 36 36 29 29 55% % 1 67 48 19 72% 2 94 54 40 57% 3 70 31 39 44% 4 106 68 38 64% 5 <6% of tickets are escalated to the engineering level CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 8/11 www.cern.ch/ i t

Inter-NOC communication Inter-NOC Intra-NOC  GGUS for LHCOPN Homegrown internal ticketing  system, y ,  E-Mail integrated with the NMS SLA tracking of the outsourced-  contracts included CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 9/11 www.cern.ch/ i t

Documentation Sharepoint collaboration workspace sites: Repository for support  entities: All procedures for Operators, Technicians Change management Change management  logging Twiki for documentation Twiki for documentation LCG Savannah for tracking LCG Savannah for tracking tasks CERN IT Department CH-1211 Genève 23 Switzerland TF-NOC - Network Operations at CERN - Gyorgy Balazs - 15.02.2011 10/11 www.cern.ch/ i t

Thank You! Q & A Q & A CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/ i t