Outline ! Control-theoretic Framework ! Service delay control on Web servers ! On-line data migration in storage servers ! ControlWare: adaptive QoS control middleware 36 #
Online Data Migration in Storage Systems ! Enterprise storage servers E-mail server; DB … " need to move data I/Os " " System expansion " Application changes data ! Always-on : e-business, migration " global data centers New device " # Online data migration " Storage system " 37 #
State of Practice Need to bound impact E-mail server; DB… " on applications! " Slow I/O’s!!! ! SAN " New device " " Migration " Script " plan data migration " storage Submover " devices " HP-UX LVM " Storage system " 38 #
The Problem ! Execute a given migration plan on-line ! Challenges Keep data consistent " Bound impact on application performance " Complete migration quickly " 39 #
Adaptive solution ! Feedback control loop: adapts migration speed based on application I/O latency " Enforce latency contract: Bounded average I/O latency " Complete migration in shortest time allowed by contract ! Standard control-theoretic design " Systematic methodology " Robust, analytically proven performance ! Handle different workloads and devices 40 #
Aqueduct Aqueduct E-mail server; DB… " migration executor " Monitor " I/Os " { L i ( k )} ! SAN " Application " { LC i } " Latency " Controller " Contract R m ( k ) " Migration " Actuator " data plan migration " storage Submover " devices " HP-UX LVM " Storage system " 41 #
Monitor Monitor " Controller " Actuator " ! Measure applications’ average I/O latency of each store in the last sampling window Current implementation: trace replayer directly monitors I/O latencies " Can interface with performance monitoring tools (HP Openview) " 42 #
Actuator Monitor " Controller " ! Fine-grained control of migration speed using HP-UX LVM Actuator " " Divide store into small (32 MB) substores (LVs) " Submover moves substore using LVM silvering Mirror " Split " Silvering " " Actuator enforces a submove rate by sleeping submv " submv " sleep " sleep " 1 submv/sw " sleep " sleep " sleep " sleep " 2 submv/sw " Sampling Window " Sampling Window " 43 #
Controller Monitor " Controller " Actuator " ! Compute error for each store i E i (k) = P*LC i - L i (k) 0<P<1: safety margin, related to burstiness k: represents the k th sampling window ! Compute worst error E min (k) = min{E i (k)} ! Integral controller computes new submove rate: R m (k) = R m (k-1) + K*E min (k) Control gain K: aggressiveness of rate change 44 #
Tuning controller parameters Victim latency VL(k) : highest Approximate linear model " average latency among all stores in VL(k+1)–VL(k)= G (R m (k)-R m (k-1)) " the k th sampling window ! Process gain G : impact of System profiling: Estimate G " submove rate on victim latency. ! Construct transfer function " ! Stability Tracking: VL(k) = P*LC ! Control Analysis " in steady state Compute K " Settling time ! Satisfy " 45 #
Experimental setup FC-60 disk array " Aqueduct " (1.05 TB, 5 RAID5 Logical Units) " HP-UX 11 & LVM " LU 0 " HP 9000-N4000 Server " emails " metadata " 8 440MHz processors " emails " metadata " Openmail " I/O Trace " Fibre Channel " LU new " ! Enterprise-scale storage server 46 #
Experiments ! Baselines: no sleeping between (sub)moves Whole-store: move one store at a time " Sub-store: move one substore at a time " ! Constant: steady Poisson streams Replace Logical Unit; migrate three 640-MB stores. " ! Openmail: trace of an enterprise e-mail server running HP Openmail Add Logical Unit; migrate a 1854 MB store and a 96 MB store " 47 #
Measure G # Tune K Process gain G: the slope of the curves " Constant " 16 Average Victim Latency (ms) y = 1.12x + 7.55 Control gain K " 14 R 2 = 0.99 12 Constant: K = 1.09 " 10 Openmail: K = 0.36 " y = 1.41x + 5.80 R 2 = 0.98 8 Openmail " 6 0 1 2 3 4 5 6 7 Submove Rate Openmail S ynthetic Linear9(Openmail) Linear9(S ynthetic) 48 #
Openmail: victim latency Average Victim Latency (ms) " LC ! 0.8*LC ! Aqueduct Sub-store Whole-store " 49 #
Openmail: latency 45 big0*(Aqueduct) 40 35 big0*(S ub5store) Latency(ms) 30 big0*(Whole5store) 25 20 15 LC ! 10 5 0 0 10 20 Time (min) Aqueduct uniformly better than baselines, but … " 50 #
Openmail: latency & submove rate 20 20 big0 S ubmv)R ate 15 15 (submv/min) submv rate latency (ms) 10 10 5 5 0 0 0 5 10 15 20 25 T im e %(m in) " Load#highest#on#new#LU#towards#end#of#migra<on# " By#design,#submove#rate#must#be#1#or#higher# $ # controller#is#working#correctly# 51 #
Openmail: average latency Aqueduct " Sub-store " Whole-store " LC ! 52 #
Openmail: latency CDF Whole1s tore A queduct S ub1s tore No=Migration 1 91% $ 0.9 0.8 0.7 76% $ 0.6 CDF 0.5 0.4 0.3 0.2 0.1 0 1 00 1 000 1 0000 1 00000 request latency (us) 53 #
Related work ! Simpler versions of the problem Take (parts of) system offline " Migrate data in “quiet periods” " ! Silvering in Logical Volume Manager [HP-UX LVM, VxVM]: maintain data consistency, no QoS guarantees ! Proportional I/O scheduling: hard to handle unpredictability ! MS Manners: no guarantee to important tasks ! Control-theory-based: distributed visual tracking, Web servers, e-mail server, database, real-time processor scheduling ... 54 #
Summary ! Migration must be executed adaptively ! Aqueduct is neither overly aggressive Average I/O latency reduced by 76% " Contract violation ratio reduced by 78% " ! nor overly conservative Average victim latency 15% lower than latency contract " ! Future More detailed sensitivity analysis " Self-tuning controller " Multi-dimensional QoS contracts " 55 #
References ! C. Lu, G. A. Alvarez, J. Wilkes, Aqueduct: Online Data Migration with Performance Guarantees, USENIX Conference on File and Storage Technologies (FAST), 2002. G.A. Alvarez, C. Lu and J. Wilkes, Method and System for Online Data ! Migration on Storage Systems with Performance Guarantees, U. S. Patent 7,167,965, January 2007. 56 #
Outline ! Control-theoretic Framework ! Service delay control on Web servers ! On-line data migration in storage servers ! ControlWare: adaptive QoS control middleware 57 #
Adaptive QoS Control Framework guarantee " QoS Guarantee " QoS Control Software " Control Loop " Controllers " QoS Architecture " Mapping " Dynamic " System Model " Identification " Dynamic " Controller Response " Design " Specs " 58 #
ControlWare Isolate programmers from control-theoretic concerns Control " Software QoS Configuration " Control Loops " QoS contract " QoS Control Loop System ID " Controller Mapper " Composition " Design " SoftBus " Controllers " Monitors " Actuators " ControlWare Library " 59 #
ControlWare : Reference ! Case studies on Squid Web cache and Apache R. Zhang, C. Lu, T. F. Abdelzaher, J. A. Stankovic, ControlWare: a ! Middleware Architecture for Feedback Control of Software performance, ICDCS, 2002. 60 #
Control-theoretic QoS Framework Map QoS guarantees to feedback control loops ! ! Establish difference equation models for computing systems via system identification ! Build practical QoS control systems Apache Web server. " Enterprise storage server. " Avionics image transmission. " ! Develop middleware for deploying QoS control FCS/nORB, FC-ORB: Distributed real-time embedded systems. " ControlWare: Internet servers. " 61 #
Recommend
More recommend
