A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System John A. Chandy John A. Chandy Department of Electrical and Computer Engineering Department of Electrical and Computer Engineering International Workshop on Data-Aware Distributed Computing June 24, 2008
Distributed Storage • Local area network • Local area network – Spread data across multiple Spread data across multiple networked nodes networked nodes – – – Parallelism and higher throughput Parallelism and higher throughput • Wide-area network ide-area network • W – – Instead of splitting data for scalability, replicate data Instead of splitting data for scalability, replicate data for availability for availability – Improves latency – Improves latency A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Replica Placement • • Where do you put the replicas? Where do you put the replicas? – Optimization problem Optimization problem – • Minimize latency or Minimize latency or maximize availability maximize availability • • Constraints: storage capacity, load balancing • Constraints: storage capacity, load balancing – – Significant existing work Significant existing work • • Latency optimization Latency optimization – Greedy algorithm, Greedy algorithm, Qiu Qiu et al. et al. – – – HotZone HotZone, , Szymaniak Szymaniak et al. et al. » » Popularity based Popularity based – Lat-cdn Lat-cdn, , Pallis Pallis et al. et al. – » Heuristic approach Heuristic approach » A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Replica Placement – Availability optimization Availability optimization – • Van Van Renesse Renesse • – – Place replicas until desired availability is reached Place replicas until desired availability is reached • Farsite Farsite • – Hill-climbing approach to replica placement Hill-climbing approach to replica placement – • Xin • Xin et al. et al. – Takes into account bimodal availability Takes into account bimodal availability – A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Replica Placement • What ’ ’ s the problem? s the problem? • What – Existing approaches assume that objects are Existing approaches assume that objects are – completely replicated completely replicated – Full replication has significant overhead Full replication has significant overhead – – Use erasure codes instead Use erasure codes instead – • • Less overhead Less overhead • Better reliability than parity Better reliability than parity • • • Placement is much more complicated Placement is much more complicated A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem Formulation • K • K data objects data objects • N storage nodes • N storage nodes • C clients • C clients n fragments of which m Each object is split into n fragments of which m • • Each object is split into fragments must be recovered to reconstruct object fragments must be recovered to reconstruct object – m=n m=n – - no redundancy - no redundancy – – m=n m=n -1 -1 - parity - parity – m= m= 1 1 – - replication - replication A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem Formulation • • Placement problem Placement problem n of the N storage of each object on n of the N – Place Place fragments fragments of each object on storage – nodes nodes – x x jk =1 j is placed on storage node k – jk =1 fragment of object j is placed on storage node k if if fragment of object • Assignment problem • Assignment problem For each object, assign each client to m m of the n n – For each object, assign each client to of the – storage nodes where the object fragments are placed storage nodes where the object fragments are placed – – y y ijk ijk =1 =1 client i i fragment of object j j from if if client retrives retrives fragment of object from k storage node k storage node A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • Overall objective is to minimize Overall objective is to minimize average latency average latency • • Cost function is Cost function is • � � � ( ) = F X , Y y ijk � ij k � K j � N i � C A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • • Constraints: Constraints: – x x is a binary variable – is a binary variable { } x � 0,1 – y y is a binary variable – is a binary variable { } x � 0,1 k k has has n n fragments – Each object Each object fragments – � x jk = n � k j � N i requests m fragments of object k client i m fragments of object k – Each Each client requests – � � i , k y ijk = m j � N A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • Constraints: • Constraints: i should request a fragment of object k from storage i k – – Client Client should request a fragment of object from storage j only if that node stores the fragment j node only if that node stores the fragment node y ijk � x jk � i , j , k j stores the Storage allocation balancing - each node j – – Storage allocation balancing - each node stores the same number of fragments same number of fragments x jk = nK � � j N k � K j services the same number of Load balancing - each node j – Load balancing - each node services the same number of – clients clients y ijk = mCK � � � j N i � C k � K A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • 0-1 integer linear programming problem 0-1 integer linear programming problem • – CN + CN + CNK variables – CNK variables – K + CK + CNK + N + N K + CK + CNK + N + N constraints – constraints – K K and C can be in the millions and N can be – and C can be in the millions and N can be in the thousands in the thousands – Problem is too large to solve with normal Problem is too large to solve with normal – methods methods A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • • Instead of doing global placement, do a placement Instead of doing global placement, do a placement on each individual object each individual object on – – Makes more sense since it is impractical to reallocate Makes more sense since it is impractical to reallocate and replace fragments every time an fragments every time an obect obect is created is created and replace x and y independent of k Make x y k – Make and independent of – • • Will cause Will cause load imbalance as all objects will be placed load imbalance as all objects will be placed on the same nodes on the same nodes – Intermediate approach - consider only a subset of Intermediate approach - consider only a subset of – objects objects n fragments, we can insure that n • Since each Since each object has object has fragments, we can insure that • K = N each node has at least 1 fragment, by setting node has at least 1 fragment, by setting and and each n introducing new constraint introducing new constraint � � j x jk = 1 k � K A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Problem formulation • With reduced object set size, derive global With reduced object set size, derive global • P placement P placement h For each new object, calculate a hash h • For each new object, calculate a hash • based on object ID, name, contents, etc. based on object ID, name, contents, etc. • Place and assign object according to object Place and assign object according to object • h mod mod K K in placement P. h in placement P. A Generalized Replica Placement Strategy to Optimize Latency in a Wide Area Distributed Storage System International Workshop on Data-Aware Distributed Computing June 24, 2008
Recommend
More recommend
