Cr Cryst stalNet alNet Faithfully Emulating Large Production - PowerPoint PPT Presentation

Cr Cryst stalNet alNet Faithfully Emulating Large Production Networks Hongqiang Harry Liu, Yibo Zhu Jitu Padhye, Jiaxin Cao, Sri T allapragada, Nuno Lopes, Andrey Rybalchenko Guohan Lu, Lihua Yuan Micr croso soft t Resear arch h Micr croso soft t Azure 1

Reliability is vital to cloud customers I can trust clouds, right? Cloud Computing Services 2

However, cloud reliability remains elusive 3

Cloud downtime cost: • 80% reported $50k/ k/hour ur or above • 25% reported $500k/ k/hour ur or above – USA SA Today y Su Surve vey y of 200 data cen enter er manager ers Cloud availability requirement: • 82% require 99.9% (3 nines nes) or above • 42% require 99.99% (4 nines) es) or above • 12% require 99.999% (5 nines) es) or above – High h avail ilability bility survey vey over er 100 0 companie nies by Infor ormation ation Tec echn hnolo ology gy and Intel ellig ligen ence ce Corp 4

What caused these outages? 5

Network is a major root cause of outages Cloud A Cloud B Cloud C June 2017-Sep 2017 Sep 20 th , 2017 Jun 8 th , 2017 Date Aug 22 nd , 2017 DynamoDB service Service Fabric, SQL DB, asia-northeast1 region experienced Availability Maximum Downtime per Year Impact IoT Hub, HDInsight, etc. disruption in US-East a loss of network connectivity 99.99% (four nines) 52.56 minutes An incorrect network Configuration error during 99.999% (five nines) A network disruption. 5.26 minutes Root Cause configuration change. network upgrades. 3.5 hours Down Time 2 hours 1.1 hours We must prevent vent such ch outages es proactively! actively! 6

We do test network hardware and software switch ch switch ch managem agement ent config igur urat ation on softwar are Vendor Unit Feature Testbeds Tests Tests Tests These e tests s say little tle about t how they y work k in production uction • Software works differently at different scales • Software/hardware bugs in corner cases • … 7

Root causes of Azure’s network incidents Unident dentified ified(2%) (2%) Hardwar ware e ASIC driver ver So Softw twar are e Bugs(36%) (36%) Failur lures es(29% (29%) failur ures es, Bugs s in route ters, s, silent nt pack cket et middle leboxes es and drops, s, management agement tools fiber cuts, s, power failur ures es wrong g ACL polici cies es, , route e Human n Data Interval: leaki king ng, , route e Errors(6% (6%) black ckhol holes es 01/2015 – 01/2017 Configuration iguration Bugs(27% 27%) 8

Ideal tests: network validation on ac actu tual al pr production oduction configuration + software + hardware + topology 9

Copying production network is infeasible Configuration Configuration Software Software Hardware Hardware Most t cost t is from Exp xpensi ensive! ve! hardwar are Configuration Configuration Configuration Configuration switch ch Software Software Software Software Hardware Hardware Hardware Hardware Product duction ion Network A C Copy of P Product duction ion Network 10

High-fidelity production environments CUSTOMER OMER IMPACTI CTING NG NETWO WORK RK INCIDENTS ENTS IN AZURE E (2015-20 2017) 7) Configuration Configuration Configuration Unidentifie fied Software Software Software Hardware vHardware vHardware Hardwar are e Failu lures es Softwar ware Bugs Configuration Configuration Configuration Configuration Configuration Configuration >69% Software Software Software Software Software Software Hardware vHardware Hardware vHardware vHardware Hardware Human Errors Product duction ion Network An E Emulat lation ion Product ductio An E ion Emulat n lation ion Product ductio ion n Network Network with h Real l Hardwar are Config figura ration on Bugs High-fid idel elit ity y product uctio ion n envir ironm onment ents 11

CrystalNet A high-fidelity, cloud-scale network emulator 12

Overview of CrystalNet external B1 Production uction Management agement S2 S1 overl rlay ay topo T ools ls by config Virtual tual links nks Host A Operat rators software version route L1 L2 Management VM T2 T4 T1 T3 Prepare Control (Linux, Windows, etc.) Host B Host C Probing & testing traffic Monitor Orch chestrator estrator Host D 13

Challenges to realize CrystalNet • sc scalabi bilit lity to emulate large networks • fl flexibi xibilit lity to accommodate heterogeneous switches • co correct ctness ness and co cost st eff ffici ciency ency of emulation boundary 14

Emulation must scale out to multiple servers switch network 1 CPU Core X 5000 = 5000 CPU Cores You need cloud to emulate a cloud network! 15

Emulation can cross cloud boundary Publi lic c Cloud ud load Priva vate te Cloud oud balancer Internet special hardware private network 16

Challenges to realize CrystalNet • scalab labil ility ity to emulate large networks ▪ scaling out emulations transparently on multiple hosts and clouds • flexib exibilit ility to accommodate heterogeneous switches • co corr rrectne ctness ss and cost effi fficiency ciency of emulation boundary 17

Heterogenous switch software sandboxes Potential ntial switch tch sandbo boxes xes Docke ker Container iner: Efficient • Supported by all cloud providers • Virtual ual Machin hine: Several vendors only offer this option • Bare-met metal al: Non-virtualizable devices (e.g. middlebox) • Needed for hardware integration tests • 18

Management challenges by heteroginousity Management Agent B1 S2 S1 Management Agent Container VM Bare Metal Host A L1 L2 T2 T4 T1 T3 Host B Host C Management Agent 19

Management challenges by heteroginousity B1 S2 S1 Container VM Bare Metal Host A L1 L2 T2 T4 T1 T3 Host B Host C 20

Building a homogenous network layer Share network PhyNet container S2 S1 S’ namespace S1’ S2’ T L S L2 L1 Heterogenous switch Host A L2’ L1’ T3 T4 T1 T2 T4’ T3’ T2’ T1’ Management Agent Host B Host C Key idea: a: maintaining taining network ork with h a h homogeno enous us layer er of contai ainer ers • start a PhyNet container for each switch • build overlay networks among PhyNet containers • Managing overlay networks with in PhyNet containers 21

Challenges to realize CrystalNet • scala labilit bility to emulate large networks ▪ scaling out emulations transparently on multiple hosts and clouds • fl flexibil ibility ity to accommodate heterogeneous devices ▪ Introducing a homogeneous PhyNet layer to open and unify network name space of devices • correctness ctness and cost t eff fficie iency ncy of emulation boundary 22

A transparent boundary is needed • We cannot extend the emulation to the whole Internet • cost • hard to get software or policy beyond our administrative domain A t trans nspar parent ent boundar dary: No sense of the existence of a boundary • C1 C1 C2 C2 Core Network & Internet (non-emulate emulated) Behaving identically as real networks • Data Center Network S1 S2 (emulate ated) L1 L2 L3 L4 L5 L6 T1 T3 T5 T2 T4 T6 23

CrystalNet constructs static boundaries routing Static ic speake aker devic ices es: information terminate the topology • maintain connections with emulated devices • customizable initial routes to emulation • C1 C2 Core Network & Internet (non-emulate emulated) no reaction to dynamics inside emulation no • Correctness? 0.0.0.0/0 Data Center Network S1 S1 S2 S2 (emulate ated) L1 L1 L2 L2 L3 L3 L4 L4 L5 L5 L6 L6 T1 T1 T3 T3 T5 T5 T2 T2 T4 T4 T6 T6 24

An example of an unsafe boundary A unsafe boundary S1 S2 Add 10.1.0.0/16 L1 L2 L3 L4 L5 L6 T1 T3 T5 T2 T4 T6 25

A proven safe boundary The boundary is a single AS, announcements never return A proven safe boundary AS100 S1 S2 AS200 AS300 L1 L2 L3 L4 L5 L6 T1 T2 T3 T4 T5 T6 Add 10.1.0.0/16 Se See paper r for proofs s and safe e boundar ary y for OSP SPF, IS-IS, S, etc. 26

Small boundaries significantly reduce cost AS100 S1 S2 AS100 S1 S2 AS200 AS300 AS200 AS300 L1 L2 L3 L4 L5 L6 L1 L2 L3 L4 L5 L6 T1 T2 T3 T4 T5 T6 T1 T2 T3 T4 T5 T6 Emulating individual PodSet Emulating Particular Layers Cost savin vings gs from Emulatin lating g the entir ire e DC: 96%~9 ~98% 8% (S (See the paper r for the algorit orithm) hm) 27

Case study 28

Shifting to regional backbones Regional Backbone Core Backbone Regional Backbone Good news: Significantly better • performance for intra- region traffic once the migration is finished Bad news: DC-1 DC-2 DC-4 DC-3 It is difficult to achieve • US-EAST US-WEST this migration without user impact 29