Visualization of Petascale Particle Data in Nvidia DGX-1 Benjamin - PowerPoint PPT Presentation

Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1 Benjamin Hernandez, PhD hernandezarb@ornl.gov Advanced Data and Workflows Group Oak Ridge Leadership Computing Facility Oak Ridge National Laboratory ORNL is managed by UT-Battelle for the US Department of Energy

Oak Ridge Leadership Computing Facility (OLCF) • Mission: Provide the computational and data resources required to solve the most challenging problems. • Highly competitive user allocation programs (INCITE, ALCC). • Projects receive 10x to 100x more resource than at other generally available centers. • We partner with users to enable science & engineering breakthroughs. 2 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Sight: Exploratory Visualization of Scientific Data • Parallel I/O & Data Staging • Client/Server architecture – “Pluggable” for in -situ to provide high end visualization visualization in laptops, desktops, and powerwalls. • Lightweight tool – Load your data • Heterogeneous scientific – Perform exploratory analysis visualization – Visualize/Save results – Take advantage of both CPU & GPU resources within a node: DGX-1 use case. – Advanced shading to enable new insights into data exploration. 3 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Sight System Architecture (in progress) Visualization Compression Local/Parallel File System Frames ADIOS I/O System HPC Cluster VTK-m VTK-m Nvidia OSPray Optix CPU Websockets Multi-GPU cores Server (DGX-1 or HTML Client multiGPU node) Server (DGX-1) Server (DGX-1) *OSPray and Nvidia Optix are finely tuned libraries for Ray Tracing in multicore and manycore architectures 4 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

ADIOS • ADIOS is an I/O framework – Provides multiple methods to stage data to a staging area (on node, off node, off machine) – Data output can be anything one wants • Different methods allow for different types of data movement, aggregation, and arrangement to the storage system or to stream over the local-nodes, LAN, WAN – It contains our own file format if you choose to use it (ADIOS-BP) – Compress/decompress data in parallel – Contains mechanisms to index and query data 5 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

First Approach: OpenGL F.S. V.S. G.S. VBO Sphere gen. Apply transfer Quads w/tex. (points) and Shading function coords 6 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

OpenGL Bindless Graphics Initialization Address pointer Display Vertices start from vboAddress to vboAddress + sizeof (float)*size 7 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Fragment Shader sphere Generation • Sphere equation: (-1,1) (1,1) 𝑠 2 = 𝑦 − 𝑦 0 2 + 𝑧 − 𝑧 0 2 + 𝑨 − 𝑨 0 2 r = 1.0, z = 1.0 x = texCoord.x y = texCoord.y zz = 1.0 – x*x – y*y if (zz <= 0.0) // removes fragments outside discard; // scale to the desired radius (-1,-1) (1,-1) // calculate diffuse illumination 8 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Results 9 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

OpenGL Multi-GPU Rendering • One MPI task for each device – Easy to implement – Each device initialize its GLX/EGL context • Multi-threading. One thread per device – In EGL is possible: • Create the main context in the main thread: mainCtx = eglCreateContext(display, config, 0, contextAttrs) • Each additional thread create a shared context: lclThrdCtx = eglCreateContext(display, config, mainCtx , contextAttrs); • Implement some mutex/semaphores to sync any updates – Vulkanize your viz ! • Devices are aware of other devices and can coordinate between each other • That’s precisely NVIDIA Optix can do 10 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Second Approach Nvidia Optix Ray Tracing Engine • “The OptiX API is an application framework for achieving optimal ray tracing performance on the GPU. It provides a simple, recursive, and flexible pipeline for accelerating ray tracing algorithms.” • “Similar to OpenGL in doing the “heavy lifting” of ray tracing and leaving capability and technique to the developers” – Plus it can use all GPUs available in your system – Naturally fits material appearance and scene illumination 11 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Nvidia Optix Programming Model • Optix provides eight programmable components, some of them are: 1. Ray generation 1 2. Intersection 3. Shading (closest hit) Shadows (any hit) 2 Selector 3 • Shaders are CUDA like syntax 12 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Nvidia Optix Graph Nodes • Geometry is defined by Graph nodes. • A tree-like hierarchy where: – Nodes at the bottom describes geometric objects. – Nodes at the top describes collections of geometric objects. Group Acceleration Transform Selector Geometry Geometry Geometry Acceleration Acceleration Group Group Group Geometry Geometry Geometry Geometry Instance Instance Instance Instance Geometry Geometry Geometry Geometry 13 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Nvidia Optix Graph Nodes • “Keep the hierarchy as flat as possible…” Group Acceleration Geometry Acceleration Group Geometry Geometry Geometry Geometry Acceleration Acceleration Acceleration Group Group Group Instance Geometry Geometry Geometry Particles Instance Instance Instance Particles Particles Particles But not too flat! 14 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Results Test Systems • Workstation – CPU Intel Xeon 20 cores… 512 GB – GPU Titan Z (2 Geforce Kepler GPU, 2x6 GB VRAM), – Ubuntu 16, Nvidia Driver • Rhea Node – CPU Intel Xeon … – GPU 2x Tesla K80 (4 Tesla Kepler GPU, 2x24 GB VRAM) – Redhat 7. Nvidia Driver All systems: • DGX-1 CUDA 8.0 – CPU Intel… Optix 4.0.2 – GPU 8x Tesla Pascal SMX, 8x16 GB VRAM Acceleration Structure: Trbvh Image resolution: 1080p – Ubuntu 16, Nvidia Driver … Shading: Phong Illumination & Ambient Occlusion 15 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Results. How fast is built the acceleration structure ? lower is better 1958 1616 1474 980 808 736 1000 108 85 80 Time (ms) 100 10 1 Workstation Rhea Node DGX-1 1 Million 10 Million 20 Million Particles 16 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Results Performance, lower is better Frame rate (worst case) Frame rate (best case) 275 250 30 fps 30 225 ms per frame 25 200 ms per frame 175 20 150 15 125 60 fps 100 10 75 50 5 25 30 fps 0 0 60 fps Workstation Rhea node DGX-1 Workstation Rhea node DGX-1 30 59 127 600 30 59 127 600 Million particles Million particles 17 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Results 18 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Discussion • DGX-1 can handle particle systems up to 10x larger in our test environment. • For particle systems of the same size DGX-1 is 10x faster than the workstation system and 4.6x faster than Rhea node – We expect for larger image resolutions DGX-1 speed up will increase. • Our preliminary tests showed DGX-1 has enough compute power to drive a powerwall – 3840 x 1080 @ 60 – 120 fps • Test larger resolution – Researchers usually are happy when they can explore datasets even at 1 fps 19 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Discussion • Nvidia Optix provides multi-GPU support with no hassle – Test if Nvidia Optix leaves free resources for analysis tasks. • Paging was removed in Optix 4.x – DGX-1 includes 40 CPU cores and 512 RAM – Using Nvidia Optix & OSPRay library will allow full system allocation to handle larger systems. • Summit is likely to support EGL through the Nvidia GPU Drivers (do not take it as a fact or alternative fact neither!) . – Best if (pre)exascale visualization tools are 100 % CUDA compliant. 20 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1

Questions ? Benjamin Hernandez, PhD hernandezarb@ornl.gov Advanced Data and Workflows Group Oak Ridge Leadership Computing Facility Oak Ridge National Laboratory Acknowledgements: Dylan Lacewell and the Nvidia Optix Team for their technical support. Datasets provided by Cheng-Yu Shi and Leonid Zhigilei from the Computational Materials Group at University of Virginia. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. 21 Exploratory Visualization of Petascale Particle Data in Nvidia DGX-1