Neural Acceleration for GPU Throughput Processors Hardik Sharma - PowerPoint PPT Presentation

Neural Acceleration for GPU Throughput Processors Hardik Sharma Jongse Park Amir Yazdanbakhsh Pejman Lotfi-Kamran * Hadi Esmaeilzadeh Alternative Computing Technologies (ACT) Lab Georgia Institute of Technology * The Institute for Research in Fundamental Sciences NGPU SM SM Neurally Accelerated GPU SM SM

Approximate computing Embracing imprecision Relax the abstraction of “ near perfect” accuracy in Data Processing Storage Communication Accept imprecision to improve performance energy dissipation resource utilization efficiency

Opportunity Many GPU applications are amenable to approximation Augmented Machine Reality SM SM Learning Computer Sensor Vision Processing SM SM Robotics Multimedia

Opportunity 100%# 90%# Approximable 80%# Non-Approximable 70%# Runtime 60%# 50%# 40%# 30%# 20%# 10%# 0%# binariza'on) blackscholes) convolu'on) inversek2j) jmeint) laplacian) meanfilter) newton9raph) sobel) sram) gmean) More than 55% of application runtime and energy is in neurally approximable regions

Neural Transformation for GPUs Neural Neural Network Network

Challenges core core core core core core core core core core core core core core core core Many core core core core core core core core Core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core core

Challenges core core core core core core core core core core core core core core core core Many Simple core core core core core core core core Core In Order core core core core core core core core core core core core core core core core Data-level core core core core core core core core Parallelism core core core core core core core core core core core core core core core core

Challenges core core core core core core core core core core core core core core core core Many Simple core core core core core core core core Core In Order core core core core core core core core core core core core core core core core Data-level core core core core core core core core Parallelism core core core core core core core core core core core core core core core core

Challenges core core core core core core core core core core core core core core core core Many Simple core core core core core core core core Core In Order core core core core core core core core core core core core core core core core Data-level core core core core core core core core Parallelism core core core core core core core core core core core core core core core core Augmenting the CPU based neural processing units to each SIMD lane imposes 31.2% area overhead

dst_reg src_reg2 src_reg1 src_reg3 NGPU Neurally-Accelerated GPU Architecture Memory Interconnection Network Partition L2 Decode Fetch Cache Issue Operand Write Collection back Active Mask SIMD Lane I-$ Off-chip SIMT DRAM Stack LSU Pipeline D-$ Streaming Multiprocessor (SM)

Neuronal Network Operations ... ... y j = x j,i x j, 0 x j,n sigmoid ( w j, 0 w j,i w j,n w j, 0 × x j, 0 + w j, 0 . . . w j,i × x j,i + . . . w j,n × x j,n + ) y j 11

Input FIFO Output FIFO Weight FIFO NGPU Neurally-Accelerated GPU Architecture 3 1 6 Acc Reg 5 Controller Sig. Unit 4 2 SIMD Lane

Output FIFO Input FIFO Weight FIFO NGPU Neurally-Accelerated GPU Architecture 3 1 6 Acc Reg 5 Controller Sig. Unit 4 2 SIMD Lane NGPU reuses the existing ALU in each SIMD lane

Output FIFO Input FIFO Weight FIFO NGPU Neurally-Accelerated GPU Architecture 3 1 6 Acc Reg 5 Controller Sig. Unit 4 2 SIMD Lane Weight FIFO is shared among all the SIMD lanes

Input FIFO Output FIFO Weight FIFO NGPU Neurally-Accelerated GPU Architecture 3 1 6 Acc Reg 5 Controller Sig. Unit 4 2 SIMD Lane Overall NGPU has ≤1% area overhead

NGPU Execution Model ld.global %r0, [addr0]; in 0 (%r 0 ) in 1 (%r 1 ) ld.global %r1, [addr1]; w0 w1 w2 w3 send.n_data %r0; n0 n1 send.n_data %r1; w4 w5 n2 recv.n_data %r2; st.global [addr2], %r2; out 0 (%r 2 ) Neurally Accelerated Neural Network GPU Application

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; …

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … SIMD lanes are in normal mode and performs precise computation

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … SIMD lanes enter neural mode

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … SIMD starts the calculation of the neural network

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … The neurally accelerated SIMD lanes autonomously calculate the neural outputs in lock-step

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … The accelerated SIMD lanes autonomously calculate the neural outputs in lock-step

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … The accelerated SIMD lanes autonomously calculate the neural outputs in lock-step

NGPU Execution Model ld.global %r0, [addr0]; … ld.global %r1, [addr1]; … send.n_data %r0; ( in 0 , in 0 , …, in 0 ) in 0 (%r 0 ) in 1 (%r 1 ) send.n_data %r1; ( in 1 , in 1 , …, in 1 ) recv.n_data %r2; w 0 × ( in 0 , in 0 , …, in 0 ) w0 w1 w2 w3 + w 2 × ( in 1 , in 1 , …, in 1 ) n0 n1 sigmoid … w 1 × ( in 0 , in 0 , …, in 0 ) w4 w5 + w 3 × ( in 1 , in 1 , …, in 1 ) n2 sigmoid … w 4 × ( n 0 , n 0 , …, n 0 ) out 0 (%r 2 ) + w 5 × ( n 1 , n 1 , …, n 1 ) sigmoid … ( out 0 , out 0 , …, out 0 ) st.global [addr2], %r2; … The accelerated SIMD lanes autonomously calculate the neural outputs in lock-step