Distributed Storage Allocation for High Reliability Derek Leong 1 - PowerPoint PPT Presentation

Distributed Storage Allocation for High Reliability Derek Leong 1 Alex Dimakis 2 Tracey Ho 1 1 Department of Electrical Engineering California Institute of Technology Pasadena, California, USA 2 Department of Electrical Engineering University of Southern California Los Angeles, California, USA ICC 2010 2010-05-26

Introduction Motivating Example A Water Analogy Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example A thirsty lion wanders across the savanna in search of water ... Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example ... he needs at least 1 ℓ of water to survive ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example ... suddenly, he finds five abandoned jerrycans ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example ... he chooses three jerrycans at random and starts chewing them open ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Budget of 1.5 ℓ Survival Allocation Probability 1 A 1 0 0 0 ? 2 1 1 1 1 1 B ? 3 3 3 3 6 1 1 1 C ? 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation A 1 � 3-Subset 1 0 0 0 ≥ 1? 2 1 ✧ (i) 1 0 0 0 2 1 ✧ (ii) 1 0 0 0 2 1 ✧ (iii) 1 0 0 0 2 1 ✧ (iv) 1 0 0 0 2 1 ✧ (v) 1 0 0 0 2 1 ✧ (vi) 1 0 0 0 2 1 ✪ (vii) 1 0 0 0 2 1 ✪ (viii) 1 0 0 0 2 1 ✪ (ix) 1 0 0 0 2 1 ✪ (x) 1 0 0 0 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation A 1 � 3-Subset 1 0 0 0 ≥ 1? 2 1 ✧ (i) 1 0 0 0 2 1 ✧ (ii) 1 0 0 0 2 1 ✧ (iii) 1 0 0 0 2 60% 1 ✧ (iv) 1 0 0 0 2 1 ✧ (v) 1 0 0 0 Survival 2 1 ✧ (vi) 1 0 0 0 Probability 2 1 ✪ (vii) 1 0 0 0 2 1 ✪ (viii) 1 0 0 0 2 1 ✪ (ix) 1 0 0 0 2 1 ✪ (x) 1 0 0 0 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation B 1 1 1 1 1 � 3-Subset ≥ 1? 3 3 3 3 6 1 1 1 1 1 ✧ (i) 3 3 3 3 6 1 1 1 1 1 ✧ (ii) 3 3 3 3 6 1 1 1 1 1 ✪ (iii) 3 3 3 3 6 1 1 1 1 1 ✧ (iv) 3 3 3 3 6 1 1 1 1 1 ✪ (v) 3 3 3 3 6 1 1 1 1 1 ✪ (vi) 3 3 3 3 6 1 1 1 1 1 ✧ (vii) 3 3 3 3 6 1 1 1 1 1 ✪ (viii) 3 3 3 3 6 1 1 1 1 1 ✪ (ix) 3 3 3 3 6 1 1 1 1 1 ✪ (x) 3 3 3 3 6 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation B 1 1 1 1 1 � 3-Subset ≥ 1? 3 3 3 3 6 1 1 1 1 1 ✧ (i) 3 3 3 3 6 1 1 1 1 1 ✧ (ii) 3 3 3 3 6 1 1 1 1 1 ✪ (iii) 3 3 3 3 6 40% 1 1 1 1 1 ✧ (iv) 3 3 3 3 6 1 1 1 1 1 ✪ (v) Survival 3 3 3 3 6 1 1 1 1 1 ✪ (vi) Probability 3 3 3 3 6 1 1 1 1 1 ✧ (vii) 3 3 3 3 6 1 1 1 1 1 ✪ (viii) 3 3 3 3 6 1 1 1 1 1 ✪ (ix) 3 3 3 3 6 1 1 1 1 1 ✪ (x) 3 3 3 3 6 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation C 1 1 1 � ≥ 1? 3-Subset 0 0 2 2 2 1 1 1 ✧ (i) 0 0 2 2 2 1 1 1 ✧ (ii) 0 0 2 2 2 1 1 1 ✧ (iii) 0 0 2 2 2 1 1 1 ✧ (iv) 0 0 2 2 2 1 1 1 ✧ (v) 0 0 2 2 2 1 1 1 ✪ (vi) 0 0 2 2 2 1 1 1 ✧ (vii) 0 0 2 2 2 1 1 1 ✧ (viii) 0 0 2 2 2 1 1 1 ✪ (ix) 0 0 2 2 2 1 1 1 ✪ (x) 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Allocation C 1 1 1 � ≥ 1? 3-Subset 0 0 2 2 2 1 1 1 ✧ (i) 0 0 2 2 2 1 1 1 ✧ (ii) 0 0 2 2 2 1 1 1 ✧ (iii) 0 0 2 2 2 70% 1 1 1 ✧ (iv) 0 0 2 2 2 1 1 1 ✧ (v) 0 0 Survival 2 2 2 1 1 1 ✪ (vi) 0 0 Probability 2 2 2 1 1 1 ✧ (vii) 0 0 2 2 2 1 1 1 ✧ (viii) 0 0 2 2 2 1 1 1 ✪ (ix) 0 0 2 2 2 1 1 1 ✪ (x) 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Budget of 1.5 ℓ Survival Allocation Probability 1 A 1 0 0 0 60% 2 1 1 1 1 1 B 40% 3 3 3 3 6 1 1 1 C 70% 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Water ≈ Coded Data Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Water ≈ Coded Data Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Motivating Example Water ≈ Coded Data storage node 1 x 1 t 1 storage node 2 x 2 r 1 s ... ... storage node n t 2 x n r 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

General Problem Description Access by a Data Collector A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the storage node 1 x 1 assumed access model or failure model storage node 2 By using an appropriate code, successful x 2 s t r recovery occurs when the total amount of ... ... storage node n data in the accessed nodes is at least the x n size of the original data object, i.e. �   ≥ 1  ∈ r Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

General Problem Description Access by a Data Collector A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the storage node 1 x 1 assumed access model or failure model storage node 2 By using an appropriate code, successful x 2 s t r recovery occurs when the total amount of ... ... storage node n data in the accessed nodes is at least the x n size of the original data object, i.e. �   ≥ 1  ∈ r Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26