In-house management tools TF-NOC George Kargiotakis - PowerPoint PPT Presentation

http://www.grnet.gr GRNET NOC In-house management tools TF-NOC George Kargiotakis (kargig@noc.grnet.gr) Andreas Polyrakis (apolyr@noc.grnet.gr) Leonidas Poulopoulos (leopoul@noc.grnet.gr)

TF-NOC 2 GRNET NOC in-house management tools

Why??? • Automation  Time  PEBKACs  Share • “Where there an API there is a way” • No API ???  Try to “create” one TF-NOC 3 GRNET NOC in-house management tools

Tools • MuPy • DCmon • Optical Network Mon • Servermon TF-NOC 4 GRNET NOC in-house management tools

MUPY TF-NOC 5 GRNET NOC in-house management tools

Why? • Munin: POWERFUL • Deployed in GRNET • Server monitoring BUT • No AAI scheme • No info aggregation • No groupings • No saved queries “Show all disk IOs graphs” ( http://munin.ping.uio.no/) “Show disk graphs from gud and knuth ” TF-NOC 6 GRNET NOC in-house management tools

What? • Parser • BeautifulSoup lib • Wrapper • Python Django • Saved queries • Open source https://code.grnet.gr/projects/mupy • Demo: http://mupy-demo.grnet.gr/ TF-NOC 7 GRNET NOC in-house management tools

DATACENTER POWER MONITORING TF-NOC 8 GRNET NOC in-house management tools

Datacenter Power-monitoring • Datacenter Ministry of Education • Appliance • Collects data • Graphs • Weekly • FTP • CSV TF-NOC 9 GRNET NOC in-house management tools

Why? • Appliance web server is slooooowwww • Live queries cause load • No combination of metrics • No PUE calculation • No public web interface • But • Provides data in html table/csv • Every 5 mins TF-NOC 10 GRNET NOC in-house management tools

How? • HTML results parser • Store nodes/metrics in db • Store data in rrd files • Combine data to calculate PUE • Provide web interface • Beta internal • Source code soon available TF-NOC 11 GRNET NOC in-house management tools

Sneak Peek-Dashboard TF-NOC 12 GRNET NOC in-house management tools

Sneak Peek-Nodes TF-NOC 13 GRNET NOC in-house management tools

Sneak Peek-Graphs TF-NOC 14 GRNET NOC in-house management tools

OPTICAL NETWORK MONITORING TF-NOC 15 GRNET NOC in-house management tools

Optical Network Monitoring • Alcatel 1626LM, 1696MS • 3 months ago: • Alcatel NMS (PM module) • No attenuation (manual) • No triggers • Fiber Cuts • Manual filters(Alarm Surveillance) • Manual triggers TIME TIME TIME TIME TF-NOC 16 GRNET NOC in-house management tools

Alcatel tools • Veeeryyy old UI • No graphing TF-NOC 17 GRNET NOC in-house management tools

Optical NetAdmin … 3 months ago TF-NOC 18 GRNET NOC in-house management tools

Requirements • Harvest • Element info • Performance data • Topology map • Graphs • Alerting • Visual, Mail • Open data • Web TF-NOC 19 GRNET NOC in-house management tools

Architecture Optical IOO client Network RRD RRD Django Analyser Memcached NMS Bootstrap UI GENOS IOO API Element Info Attenuation Attenuation Fiber Cuts Topology Element Graphs Diff TF-NOC 20 GRNET NOC in-house management tools

Optical Mon • http://mon.grnet.gr (Optical Network Mon) • IOO Client: http://code.grnet.gr • Source code soon open • Open access • Frequent feature updates TF-NOC 21 GRNET NOC in-house management tools

Anomaly detection • On RRD files • Attenuation (diff of counters) • Iterate measurements • Pattern matching • If att==0 for x minutes then fiber_cut • If |att_diff|>a for x minutes then anomaly • More simple yet smart checks • Show visual alerts • Send mail to optnetadmins TF-NOC 22 GRNET NOC in-house management tools

Anomaly detection improvements • RRD Holt-Winters Time Series Forecasting TF-NOC 23 GRNET NOC in-house management tools

Optical NetAdmin … now TF-NOC 24 GRNET NOC in-house management tools

SERVERMON TF-NOC 25 GRNET NOC in-house management tools

Sneak Peek TF-NOC 26 GRNET NOC in-house management tools

Sneak Peek TF-NOC 27 GRNET NOC in-house management tools

Sneak Peek TF-NOC 28 GRNET NOC in-house management tools

Sneak Peek TF-NOC 29 GRNET NOC in-house management tools

Source Code • https://code.grnet.gr/ TF-NOC 30 GRNET NOC in-house management tools

Thank you Questions ? https://twitter.com/grnetnoc https://www.facebook.com/noc.grnet.gr TF-NOC 31 GRNET NOC in-house management tools