CLUSTAR: AI Training Platform Powered by High Performance Networking - PowerPoint PPT Presentation

CLUSTAR: AI Training Platform Powered by High Performance Networking Junxue ZHANG EVP CLUSTAR PhD SING Lab, HKUST AGUEST 1,2018

Deep Learning Is Becoming Increasingly Important Computer Vision Natural Language Processing Auto-driving Cars 27

How does Deep Learning Work ? 𝑧 = 𝑏 ∗ 𝑦 + 𝑐 𝒚 𝒛 𝒛 𝒒𝒔𝒇𝒆 1 5 mini batch 2 7 𝑐 = 1 1 𝑡𝑣𝑛 𝑏 = 1 𝑦 Input Layer Output Layer 28

How does Deep Learning Work ? 𝑧 = 𝑏 ∗ 𝑦 + 𝑐 𝒚 𝒛 𝒛 𝒒𝒔𝒇𝒆 1 5 2 mini batch Forward Pass 3 2 7 𝑐 = 1 1 𝑡𝑣𝑛 𝑏 = 1 𝑦 Input Layer Output Layer 29

How does Deep Learning Work ? 𝑧 = 𝑏 ∗ 𝑦 + 𝑐 𝒚 𝒛 𝒛 𝒒𝒔𝒇𝒆 1 5 2 mini batch Forward Pass 3 2 7 ; = 1 𝑀 = 𝐷 4 𝑧 − 𝑧 6789 2 4 𝑧 − 𝑧 6789 𝑐 = 1 1 Calculating Loss 𝑡𝑣𝑛 𝑏 = 1 𝑦 Input Layer Output Layer 30

How does Deep Learning Work ? 𝑧 = 𝑏 ∗ 𝑦 + 𝑐 𝒚 𝒛 𝒛 𝒒𝒔𝒇𝒆 1 5 2 mini batch Backpropagation 3 2 7 ; = 1 𝑀 = 𝐷 4 𝑧 − 𝑧 6789 2 4 𝑧 − 𝑧 6789 𝑐 = 1 − 0.1 ∗ −7 = 1.7 1 × 𝜖𝑧 6789 𝜖𝑀 𝜖𝑀 Calculating Loss 𝜖𝑏 = = 4 𝑧 6789 − 𝑧 𝑦 = −11 𝜖𝑧 6789 𝜖𝑏 𝑡𝑣𝑛 𝜖𝑀 𝜖𝑀 × 𝜖𝑧 6789 𝜖𝑐 = = 4 𝑧 6789 − 𝑧 = −7 𝑏 = 1 − 0.1 ∗ −11 = 2.1 𝜖𝑧 6789 𝜖𝑐 𝑦 𝑏 = 𝑏 − 𝑠 𝜖𝑀 𝑐 = 𝑐 − 𝑠 𝜖𝑀 𝜖𝑏 𝜖𝑐 Input Layer Output Layer 31

How does Deep Learning Work ? 𝑧 = 𝑏 ∗ 𝑦 + 𝑐 𝒚 𝒛 𝒛 𝒒𝒔𝒇𝒆 3 9 Backpropagation 5 13 ; = 1 𝑀 = 𝐷 4 𝑧 − 𝑧 6789 2 4 𝑧 − 𝑧 6789 𝑐 = 1 − 0.1 ∗ −7 = 1.7 1 × 𝜖𝑧 6789 𝜖𝑀 𝜖𝑀 Next Calculating Loss 𝜖𝑏 = = 4 𝑧 6789 − 𝑧 𝑦 = −11 Iteration 𝜖𝑧 6789 𝜖𝑏 𝑡𝑣𝑛 𝜖𝑀 𝜖𝑀 × 𝜖𝑧 6789 𝜖𝑐 = = 4 𝑧 6789 − 𝑧 = −7 𝑏 = 1 − 0.1 ∗ −11 = 2.1 𝜖𝑧 6789 𝜖𝑐 𝑦 𝑏 = 𝑏 − 𝑠 𝜖𝑀 𝑐 = 𝑐 − 𝑠 𝜖𝑀 𝜖𝑏 𝜖𝑐 Input Layer Output Layer 32

How does Deep Learning Work ? Input Layer Output Layer Hidden Layer 33

How does Deep Learning Work ? Backpropagation Backpropagation Backpropagation D 𝑥 E; D 𝑥 ;C D 𝑥 FE Calculating Loss Forward Pass Forward Pass Forward Pass Input Layer Output Layer Hidden Layer 34

The Big Data Drives a New Paradigm for Training 1. Data is too large to fit in single machine 2. The training time is too long Uber: it usually takes weeks or longer to complete [1] 35

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Networking Data Data Partition 1 Partition 2 Worker 1 Worker 2 36

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Pull Parameters From Servers Networking 𝑥 E 𝑥 ; 𝑥 E 𝑥 ; Data Data Partition 1 Partition 2 Worker 1 Worker 2 37

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Networking 𝑥 E 𝑥 ; 𝑥 E 𝑥 ; Data Data Input Input Partition 1 Partition 2 Worker 1 Worker 2 Forward Pass Forward Pass 38

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Networking Calculating Loss Calculating Loss 𝑥 E 𝑥 ; 𝑥 E 𝑥 ; Data Data Input Input Partition 1 Partition 2 Worker 1 Worker 2 Forward Pass Forward Pass 39

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Networking DD DD 𝑥 E D 𝑥 ; D 𝑥 E 𝑥 ; Data Data Partition 1 Partition 2 Worker 1 Worker 2 Backpropagation Backpropagation 40

Networking Plays an Important Role … 𝑥 E 𝑥 ; Parameter Server Push parameters to Servers Networking DD DD 𝑥 E D 𝑥 ; D 𝑥 E 𝑥 ; Data Data Partition 1 Partition 2 Worker 1 Worker 2 Backpropagation Backpropagation 41

Networking Plays an Important Role Networking is critical to performance ! … 𝑥 E 𝑥 ; Parameter Server Push parameters to Servers Networking DD DD 𝑥 E D 𝑥 ; D 𝑥 E 𝑥 ; Data Data Partition 1 Partition 2 Worker 1 Worker 2 Backpropagation Backpropagation 42

Networking Plays an Important Role The speedup achieved after utilizing the 40Gbps networking bandwidth with CLUSTAR Model Logistic Multi-layer Alexnet VGG-16 Resnet-50 Regression perceptron Speedup 2.59x 3.45x 1.6x 1.33x 1.03x 43

CLUSTAR: AI Training Platform Powered by High Performance Networking The important of networking towards AI system equals the traffic system towards cities Between 2 Machines Multiple Machines AI Protocol Wider Roads Traffic Scheduling New Traffic Rule for AI Key Technology （ World-leading Research Achievements ） GDR ParaExpress MLT Smart Networking Scheduling • Utilize the SGD of AI training • Towards 0-copy data flow • Resilient and adaptive parameter aggregation • Co-flow scheduling • Utilize RDMA and GPUDirect • Semi-loss tolerance • Tackles the disadvantage of • Elephant & Mice flow • Model quality awareness • Integrated with TensorFlow Parameter Server & Ring AllReduce scheduling 44

CLUSTAR: AI Training Platform Powered by High Performance Networking The important of networking towards AI system equals the traffic system towards cities Between 2 Machines Multiple Machines AI Protocol Wider Roads Traffic Scheduling New Traffic Rule for AI Key Technology （ World-leading Research Achievements ） GDR ParaExpress MLT Smart Networking Scheduling • Utilize the SGD of AI training • Towards 0-copy data flow • Resilient and adaptive parameter aggregation • Co-flow scheduling • Utilize RDMA and GPUDirect • Semi-loss tolerance • Tackles the disadvantage of • Elephant & Mice flow • Model quality awareness • Integrated with TensorFlow Parameter Server & Ring AllReduce scheduling 45

CLUSTAR: AI Training Platform Powered by High Performance Networking The important of networking towards AI system equals the traffic system towards cities Between 2 Machines Multiple Machines AI Protocol Wider Roads Traffic Scheduling New Traffic Rule for AI Key Technology （ World-leading Research Achievements ） GDR ParaExpress MLT Smart Networking Scheduling • Utilize the SGD of AI training • Towards 0-copy data flow • Resilient and adaptive parameter aggregation • Co-flow scheduling • Utilize RDMA and GPUDirect • Semi-loss tolerance • Tackles the disadvantage of • Elephant & Mice flow • Model quality awareness • Integrated with TensorFlow Parameter Server & Ring AllReduce scheduling 46

可编程⽹罒络 Broadcom FPGA ASIC RDMA⽹罒络 Intel Nvidia AMD 寒武纪 Mellanox P4 CPU E8 Storage Clustar AI Fabrics RoCE 智能⽹罒卡 Spark优化 TensorFlow优化 容器塀编排引擎 交互编程界⾯靣 GPU 全闪存存储 件 ⾃臫动驾驶 基 础 设 施 应⽤甩 硬 ⾦金喇融⾏行降业应⽤甩 语⾳韴识别 ⾃臫然语⾔訁处理痢 计算机视觉 智能反欺诈 智能⽆旡⼈亻机 安防⾏行降业应⽤甩 互联⽹罒⾏行降业应⽤甩 制造业⾏行降业应⽤甩 医疗⾏行降业应⽤甩 ⽤甩 通 政府⾏行降业应⽤甩 CLUSTAR Platform 星 数据预处理 离线训练 在线训练 多租户管理 任务调度 运维监控 云 平 台 47

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 48

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 49

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 50

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 51

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 52

GDR: Towards Zero Copy Data Flow Socket 1 Socket 2 Socket 1 Socket 2 CPU CPU CPU CPU Memory Memory Memory Memory RDMA RDMA GPU GPU GPU GPU GPU GPU NIC NIC Server 1 Server 2 Data Center Networking The unnecessary copy between RNIC/Memory and GPU RAM/Memory enlarges latency, degrades throughput and burns CPU 53