SC 2015 Data Partitioning Strategies for Graph Workloads on Heterogeneous Clusters Michael LeBeane, Shuang Song, Reena Panda, Jee Ho Ryoo, Lizy K. John The University of Texas at Austin mlebeane@utexas.edu
SC 2015 Motivation Data Data Compute Compute Node ▪ Heterogeneity is pervasive in Node Data modern data centers [][] Data ▪ Graph analytics are a pervasive Network workload in the data center [] – Many frameworks available to Compute Compute Node efficiently and easily perform graph Node analytics [][][][] ▪ Most frameworks are not Data Compute Compute Node equipped to deal with Node heterogeneity in the data center Data 2 Michael LeBeane 11/18/2015
SC 2015 Background 1 ▪ All work performed on PowerGraph[] framework 2 ▪ Three relevant graph partitioning topics: 2 1 – Online vs. Offline Partitioning 2 2 1 – Vertex vs. Edge Cut 1 – Gather/Apply/Scatter 1 2 1 1 ▪ Online vs. Offline Partitioning 3 Michael LeBeane 11/18/2015
SC 2015 Background Machine X Machine Y (a) Vertex Cut 1 1’ 1 2 1 2 Master 2 2’ 1 2 5 F(x) 5’ 3 4 3 4 3 3’ 1 2 1 2 4 3 Ghost 4 4’ (b) Apply (a) Gather (c) Scatter 4 3 3 4 (b) Edge Cut ▪ Gather/Apply/Scatter ▪ Vertex vs. Edge Cut 4 Michael LeBeane 11/18/2015
SC 2015 Workload Skew in Heterogeneous Data Centers Barrier Compute Idle Compute Idle Fast Data Communication Communication Node Compute Compute Slow Data Communication Communication Node Time ▪ Normal Data Partitioning Barrier Runtime Compute Compute Fast Improvement Data Node Communication Communication Compute Compute Slow Data Communication Communication Node Time ▪ Skewed Data Partitioning 5 Michael LeBeane 11/18/2015
SC 2015 Heterogeneous Graph Analytics File 1 File 2 File N ▪ Local node computation time Baseline Computation Capacity Partitioner dependent on data Graph distribution Loading Files 1 2 ▪ To properly balance work, we Heterogeneity Aware Partitioning need: Partitioner Graph – Estimation of each node’s computational capacity Finalizing Graph – Partitioning algorithms that Data Data Data Data account for skewed App Execution Node 1 Node 2 Node 3 Node n computational capacity 6 Michael LeBeane 11/18/2015
SC 2015 Heterogeneous Computation Capacity File 1 File 2 File N ▪ Computation capacity is Baseline Computation Capacity Partitioner complex Graph Loading Files ▪ Dependent on many factors: 1 2 – Hardware of the node Heterogeneity Aware Partitioning Partitioner Graph – Nature of the graph – Nature of the algorithm Finalizing Graph – Communication patterns Data Data Data Data ▪ Can we determine a simple, App Execution Node 1 Node 2 Node 3 Node n static estimate? 7 Michael LeBeane 11/18/2015
SC 2015 Skew Factor Calculation Name HW Threads Memory Network Thread Skew Factor Memory Skew Factor c4.xlarge 4 7.5 GB 100 Mbps to 1.86 Gbps 1 1 c4.2xlarge 8 15 GB 100 Mbps to 1.86 Gbps 3 2 c4.4xlarge 16 30 GB 100 Mbps to 1.86 Gbps 7 4 c4.8xlarge 36 60 GB up to 8.86 Gbps 17 8 ▪ Static estimate of node computational capacity could be based on: – Threads: Logical compute threads on node (default N – 2 ) – Memory: Physical memory assigned to a node – Profiling: Local throughput of graph subset and algorithm ▪ We will refer to the estimated ratios of computation capacity as the skew factor of the heterogeneous data center 8 Michael LeBeane 11/18/2015
SC 2015 Heterogeneous Partitioning Algorithm File 1 File 2 File N ▪ Online partitioning algorithms Baseline Computation Capacity Partitioner must be modified to support Graph skew factor Loading Files 1 2 Heterogeneity Aware Partitioning Partitioner Graph ▪ Easy to modify current online partitioning algorithms Finalizing Graph Data Data Data Data ▪ We have modified 5 popular App Execution Node 1 Node 2 Node 3 Node n algorithms from multiple sources 9 Michael LeBeane 11/18/2015
SC 2015 Problem Formulation ▪ Statically estimated based on: – Threads: Logical compute threads on node (default N – 2 ) – Memory: Physical memory assigned to a node – Profiling: Local throughput of graph subset and algorithm ▪ Statically estimated based on: 10 Michael LeBeane 11/18/2015
SC 2015 Random Skewed Partitioner Edge Edge Random Skew Random Assignment Factor Assignment …. …. Node 0 Node 1 Node n Node 0 Node 1 Node n ▪ Original ▪ Skewed ▪ Random assignment of edges to nodes 11
SC 2015 Greedy Skewed Partitioner Edge Edge Skew Factor Heuristic Heuristic Assignment Assignment Balance Balance …. …. Node 0 Node 1 Node n Node 0 Node 1 Node n ▪ Original ▪ Skewed ▪ Greedy decision using current distribution of edges – Either locally or coordinated 12 Michael LeBeane 11/18/2015
SC 2015 Grid Skewed Partitioner Edge Edge Grid Hash Grid Hash Skew Grid Grid Factor Random Random Selection Selection …. …. Node 0 Node 1 Node n Node 0 Node 1 Node n ▪ Original ▪ Skewed ▪ Greedy decision using current distribution of edges – Either locally or coordinated 13 Michael LeBeane 11/18/2015
SC 2015 Hybrid Skewed Partitioner Vertex Vertex Degree > Degree > Threshold Threshold Edge Vertex Edge Vertex Skew Factor Random Heuristic Random Heuristic Skew Assignment Assignment Assignment Factor Assignment …. …. …. …. Node 0 Node 1 Node n Node n Node 0 Node 1 Node n Node 0 Node 1 Node n Node 0 Node 1 ▪ Original ▪ Skewed ▪ Random assignment of edges/verticies to nodes based on degree 14 Michael LeBeane 11/18/2015
SC 2015 Ginger Skewed Partitioner Vertex Vertex Degree > Degree > Threshold Threshold Edge Vertex Edge Vertex Skew Factor Random Heuristic Random Heuristic Skew Assignment Assignment Assignment Factor Assignment Balance Balance …. …. …. …. Node 0 Node 1 Node n Node n Node 0 Node 1 Node n Node 0 Node 1 Node n Node 0 Node 1 ▪ Original ▪ Skewed ▪ Random assignment of edges/verticies to nodes based on degree 15 15 Michael LeBeane Michael LeBeane 11/18/2015 11/18/2015
SC 2015 Experimental Setup ▪ Algorithms – Graph: PageRank (PR), Connected Components (CC), Triangle Count (TC) – Matrix: Stochastic Gradient Descent (SGD), Alternating Least Squares (ALS) ▪ Data Sets Name Vertices Edges Size (Uncompressed) Type Algorithms amazon 403,394 3,384,388 46MB Directed Graph PR,CC,TC citation 3,774,768,NA 16,518,948 268MB Directed Graph PR,CC,TC netflix NA NA 100MB Sparse Matrix ALS,SGD road-map 1,379,917 1,921,660 84MB Undirected Graph PR,CC,TC social-network 4,847,571 68,993,773 1.1GB Directed Graph PR,CC,TC twitter 41,000,000 1,400,000,000 25GB Directed Graph PR,CC,TC wiki 2,394,385 5,021,410 64MB Directed Graph PR,CC,TC 16 Michael LeBeane 11/18/2015
SC 2015 Experimental Setup ▪ Data Center – Graph: PageRank (PR), Connected Components (CC), Triangle Count (TC) – Matrix: Stochastic Gradient Descent (SGD), Alternating Least Squares (ALS) ▪ Skew Factor – Results use Thread Based Skew Factor 17 Michael LeBeane 11/18/2015
SC 2015 Execution Time 60 Transmit Receive Gather Apply Scatter 50 Baseline Skewed Runtime (s) 40 30 20 10 0 Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger social_network amazon citation road_map wiki ▪ Pagerank 18 Michael LeBeane 11/18/2015
SC 2015 Execution Time 16 160 Receive Gather Apply Scatter Transmit 14 140 Right Axis 12 120 Baseline Runtime (s) Runtime (s) 10 100 Skewed 8 80 6 60 4 40 2 20 0 0 Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger social_network amazon citation road_map wiki ▪ Connected Components 19 Michael LeBeane 11/18/2015
SC 2015 Execution Time 6 60 Right Axis Receive Gather Apply Scatter Transmit 5 50 Runtime (s) Runtime (s) 4 40 Baseline 3 30 Skewed 2 20 1 10 0 0 Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger social_network amazon citation road_map wiki ▪ Triangle Count 20 Michael LeBeane 11/18/2015
SC 2015 Execution Time 18 100 TX RX G A S TX RX G A S 16 Baseline Skewed 90 14 80 Baseline 12 70 Runtime (s) Runtime (s) 10 60 Skewed 50 8 40 6 30 4 20 2 10 0 0 Random Greedy Grid Hybrid Ginger Random Greedy Grid Hybrid Ginger netflix netflix ▪ Stochastic Gradient Descent ▪ Alternating Least Squares 21 Michael LeBeane 11/18/2015
SC 2015 Data distribution 1 Relative Edge Distribution 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 SRandom SGreedy SGrid SHybrid SGinger SRandom SGreedy SGrid SHybrid SGinger SRandom SGreedy SGrid SHybrid SGinger SRandom SGreedy SGrid SHybrid SGinger SRandom SGreedy SGrid SHybrid SGinger Non-Skew Target-Skew social_network amazon citation road_map wiki optimal Node (1) Node (3) Node (7) Node (17) ▪ Ideal distribution 17-7-3-1 22 Michael LeBeane 11/18/2015
Recommend
More recommend
