Achieving Portable Performance for GTC-P with OpenACC on GPU, - - PowerPoint PPT Presentation

▶

Aug 21, 2023 197 likes •422 views

Achieving Portable Performance for GTC-P with OpenACC on GPU, multi-core CPU, and Sunway Many-core Processor Step ephen en Wa Wang 1 , James Lin 1,4 , William Tang 2 , Stephane Ethier 2 , Bei Wang 2 , Simon See 1,3 and Satoshi

SLIDE 1

Achieving Portable Performance for GTC-P with OpenACC on GPU, multi-core CPU, and Sunway Many-core Processor

Step ephen en Wa Wang†1, James Lin†1,4, William Tang†2, Stephane Ethier†2, Bei Wang†2, Simon See†1,3 and Satoshi Matsuoka†4

†1 Shanghai Jiao Tong University, Center for HPC †2 Princeton University, Institute for Computational Science & Engineering (PICSciE) and Plasma Physics Laboratory(PPPL) †3 NVIDIA corporation †4 Tokyo Institute of Technology

GTC 2017, San Jose, USA May 11, 2017

SLIDE 2

1000 100 10 Core Number

Challenges of supporting multi- and many-cores, the territory of OpenMP

SLIDE 3

GTC-P: Gyrokinetic Toroidal Code - Princeton

Developed by Princeton to accelerate progress in highly-scalable plasma

turbulence HPC Particle-in-Cell (PIC) codes

Successfully applied to high-resolution problem-size-scaling studies relevant

to the Fusion’s next-generation International Thermonuclear Experimental Reactor (ITER).

Modern “co-design” version of the comprehensive original GTC code with

focus on using Computer Science performance modeling to improve basic PIC operations to deliver simulations at extreme scales with unprecedented resolution & speed on variety of different architectures worldwide

Includes present-day multi-petaflop supercomputers, including Tianhe-2,

Titan, Sequoia, Mira, etc., that feature GPU, CPU multicore, and many- core processors

KEY REFERENCE: W. Tang, B. Wang, S. Ethier, G. Kwasniewski, T.

Hoefler and etc. ,“Extreme Scale Plasma Turbulence Simulations on Top Supercomputers Worldwide” , Supercomputing (SC), 2016 Conference, Salt Lake City, Utah, USA

3

SLIDE 4

OpenACC Implementations

Challenges

a. Memory-bound kernels b. Data hazard c. Random memory access

Implementations

a. Increase memory bandwidth b. Use atomic operations c. Take advantage of local memory

Six Major Subroutines of GTC-P hotspots

SLIDE 5

OpenACC Implementations – present directive

SLIDE 6

OpenACC Implementations – atomic directive

SLIDE 7

Sunway many-core (SW 26010) Baseline: Serial code

n 1 MPE

OpenACC code

n 64 CPE

Elapsed Time (s) 4.7 2360.5

504x slower !!!

2x slower 201x slower!

Run the single OpenACC code base: huge performance gap on x86 and Sunway

GPU (NVIDIA K20) Baseline: CUDA OpenACC Elapsed Time (s) 7.9 16.7 x86 multicore (Intel SNB) Baseline: OpenMP OpenACC Elapsed Time (s) 7.9 1572.8 OpenMP allocates the array copy on each thread and reduce, without atomic operations.

unacceptable

SLIDE 8

Data Hazard

T1 T2 T3 T4

array[thread-id][n] - copy for T1 Reduction (Add) array[n] Irregular Memory Access (Fetch-and-Add)

Our solution for multi- and many-core: using thread-id to duplicate copies for reduction to replace the Fetch-and-Add atomic operation

array[thread-id][n] - copy for T2 array[thread-id][n] - copy for T3 array[thread-id][n] - copy for T4

SLIDE 9

Performance w/o atomic operations on x86 CPU

Thread ID is not

supported for x86 in OpenACC standard yet.

Private function in PGI

compiler is used here: __pgi_blockidx()

PGI compiler 16.10 Baseline

SLIDE 10

acc_thread_id is a customized extension provided in Sunway OpenACC

Implementation on Sunway many-core processor: a customized thread-id extension available from Sunway OpenACC

Architecture overview of SW26010

SLIDE 11

Optimization on Sunway many-core processor: data locality in 64KB Scratch Pad Memory

Using tile directive to coalesced access data

by per DMA request. The optimum tile size can take full usage of 64KB SPM.

Elapsed Time(s)

Lower is better

tile_size

Memory hierarchy of CPE

Keep data in SPM instead of global

memory access.

SPM

SLIDE 12

(*) Optimization on Sunway many-core processor

256-bit SIMD intrinsic

OpenACC code immediate code (.host and .slave)

“-keep” or “-dumpcommand” can let compiler generate it.

Execution file swacc (S2S compiler) sw5cc (native compiler)

This part in push kernel can achieve 5.6x speedup. But the cost of this kernel is

too small compared with the entire GTC-P code.

SLIDE 13

Performance on Sunway many-core processor

500 1000 1500 2000 2500

Sequential(MPE) OpenACC(CPE) +w/oatomics +Tile +SPMlibrary

Lower is better

Baseline 1.1X 2.5X

Elapsed time [sec]

Charge Push Poisson Field Smooth Shift 13

Avoid atomic operations.
Increase DMA bandwidth
Strengthen data locality

in SPM

(*) In-build SIMD code

SLIDE 14

Performance and portability of GTC-P on GPU

SLIDE 15

Use native atomic instructions on P100

Native atomic instructions

(FP64) are supported on Pascal architecture.

Compare the PTX code

generated by PGI 16.10 compiler on K80 and P100.

SLIDE 16

OpenACC version of GTC-P on K80 and P100

Performance of OpenACC

version on P100 is close to CUDA code due to the better atomic instructions support.

OpenACC benefit from the

hardware support on the latest GPU architecture.

SLIDE 17

Use specific algorithm for GPU in OpenACC code

Remove auxiliary array which use to store the 4 points

SLIDE 18

Performance results of OpenACC version with new algorithm on GPU

CUDA OpenACC new OpenACC B 100 1 2 * GPU 1399MB/GPU 3070MB/GPU 1501MB/GPU B 100 1 4 * GPU 742MB/GPU 1569MB/GPU 785MB/GPU

50% device memory usage reduce

Tesla K40 GPU

SLIDE 19