Large-scale Virtualization in the Emulab Network Testbed Mike Hibler, Robert Ricci, Leigh Stoller, Jonathon Duerig, Shashi Guruprasad, Tim Stack, Kirk Webb, Jay Lepreau
Emulab: Network Testbed 2
Emulab: Network Testbed 2
Emulab: Network Testbed 2
Emulab: Network Testbed 2
What’s Wrong? 3
What’s Wrong? Too small 3
What’s Wrong? Too small Inefficient 3
What’s Wrong? Too small Inefficient Solution? 3
What’s Wrong? Too small Inefficient Solution? Virtualize 3
What’s Wrong? Too small Inefficient Solution? Virtualize Multiplex 3
The Basic Idea:
The Basic Idea: Use virtualization to perform network experiments using fewer physical resources.
The Basic Idea: Use virtualization to perform network experiments using fewer physical resources. ... and do this in a way that:
The Basic Idea: Use virtualization to perform network experiments using fewer physical resources. ... and do this in a way that: is transparent to applications and preserves experiment fidelity
5
Challenges Opportunities 5
Challenges Opportunities • Fidelity 5
Challenges Opportunities • Fidelity • Preserve network topology 5
Challenges Opportunities • Fidelity • Closed world • Preserve network topology 5
Challenges Opportunities • Fidelity • Closed world • Can re-run • Preserve network experiments topology 5
Not Just Picking a VM Technology 6
Not Just Picking a VM Technology Complete Virtual Network Experimentation System 6
Full System • Virtualization technology • Host and network • Resource mapping • Feedback-directed emulation • IP address assignment • Scalable control system • Routing table calculation 7
<virtualization>
Start: FreeBSD jail • Namespace isolation • Virtual disks • We added network virtualization: • Ability to bind to multiple interfaces • New virtual network device ( veth ) • Separate routing tables 9
10
10
10
10
10
10
10
10
<mapping>
What does it mean to make a good mapping?
Good Mapping • Pack well • Use resources efficiently • Specifying packing criteria • Do it quickly • Critical path for creating an experiment 13
assign • Solves an NP-hard problem • Pack both nodes and links • Avoid scarce resources • Paper: [Ricci+:CCR03] • Based on simulated annealing • We extended for virtual nodes 14
Resource-Based Packing • Use quantities we can directly measure • Resource-based system “This virtual node uses 100 MHz of CPU” “This physical node has 3 GHz of CPU” • Works well for heterogenous virtual and physical nodes 15
Assigning Quickly
Small Topologies 17
Small Topologies 17
Small Topologies 17
Virtual Topologies 18
Virtual Topologies 18
Virtual Topologies 18
Prepass 19
Prepass 19
Prepass 19
Scaling With Prepass 20
Scaling With Prepass 200 20
Mapping Quality 21
Mapping Quality 21
Mapping Quality 21
Mapping Quality 21
<feedback>
How do I know how tightly I can pack my virtual nodes?
How do I know how tightly I can pack my virtual nodes? I don’t!
Closed, repeatable world
The Plan 25
The Plan • Pick a packing 25
The Plan • Pick a packing • Run experiment 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts • If artifacts found: 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts • If artifacts found: • Re-pack 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts • If artifacts found: • Re-pack • Repeat 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts • If artifacts found: • Re-pack • Repeat 25
The Plan • Pick a packing • Run experiment • Monitor for artifacts • If artifacts found: • Re-pack • Repeat 25
Picking Initial Packing • Start one-to-one • Possibly with a subset of topology • Start tightly packed • Optimistically assume low usage 26
Monitoring for Artifacts • CPU near 100% • Significant paging activity • Disk utilization 27
Re-Packing • Measure resource use • Feed into resource-based packing 28
Feedback in a Nutshell • Rely on packing, not isolation • Discover packing factors empirically • Re-use between experiments 29
<numbers>
kindex: Packing Factors
Feedback Case Study
Feedback Case Study Transactions Response Round Per Second Time (s)
Feedback Case Study Transactions Response Round Per Second Time (s) Bootstrap: 74 physical 2.29 0.43
Feedback Case Study Transactions Response Round Per Second Time (s) Bootstrap: 74 physical 2.29 0.43 Round 1: 7 physical 1.85 0.53
Feedback Case Study Transactions Response Round Per Second Time (s) Bootstrap: 74 physical 2.29 0.43 Round 1: 7 physical 1.85 0.53 Round 2: 7 physical 2.29 0.43
Deployed Use • Creation time: 7 minutes for 100 nodes • 5,125 experiments • 296,621 virtual nodes • 32% of Emulab nodes virtual • 5.75 average packing factor • Average: 58 nodes, max: 1,000 nodes 33
Conclusion • Virtualization increases Emulab’s capacity • Transparently • Preserves fidelity • Requires solving several challenging problems • Proven useful in production www.emulab.net 34
<end/>
Related Work • Virtual Network Emulation • ModelNet, DieCast • Virtual Machines • Xen, VMWare, vservers, OpenVZ, NET • Network Virtualization • NetNS, OpenVZ, Trellis, IMUNES • Feedback-based Mapping • Hippodrome 36
ModelNet • Applications can only run on edge nodes: single-homed only • More basic virtualization • No artifact detection • No feedback system • Emulab has richer control framework • Scales to much larger interior networks 37
Minimal Effective Virtualization • Application transparency • Application fidelity • System capacity 38
Application Transparency • Real applications • Simulation • Virtual machines • Keep most semantics of unshared machines • Simple processes • Preserve experimenter’s topology • Full network virtualization 39
Application Fidelity • Physical results ≈ Virtual results • Virtual node interference • Perfect resource isolation • Detect artifacts and re-run 40
System Capacity • Low overhead • In-kernel (vservers, jails) • Hypervisor (VMWare, Xen) • Don’t prolong experiments • DieCast 41
Recommend
More recommend
