Good Times with a Pile of GPUs Elizabeth Baumel About me Unity - - PowerPoint PPT Presentation

Good Times with a Pile of GPUs Elizabeth Baumel

About me ● Unity - DOTS Team ● Disbelief - Unreal games ○ Gears of War 4 - D3D12 multi-GPU support on UWP ○ Other, more exotic D3D12 multi-GPU stuff ● Sony - PS4 graphics dev support ● Intel - DirectX 11 drivers

Haha, what? ● A pile implies more than 1 ● Multiple GPUs ○ ...you might say…….. ● multi-GPU :)

What is Multi-GPU?? ● Using multiple GPUs in a single machine to do more work. ○ Rendering ○ Compute ○ Games/real-time rendering and simulation ● NOT about: ○ GPU supercomputing clusters ○ Cryptocurrency mining

History of Multi-GPU ● 1990 ● Silicon Graphics SkyWriter ○ Dual pipeline ■ Dual screen ○ Hyperpipeline ■ 2 GPUs, 1 display ● Alternate frame rendering ○ the OG AFR!! Source: https://web.archive.org/web/20110715174342/http://www.reputable.com/~s kywriter/skywriter/techreport/6.html

History of Multi-GPU ● 1998 ● 3dfx Voodoo2 ● SLI ○ S can- L ine I nterleave ○ Each card rendered alternating scan lines ● Higher max resolution ○ 1024x768 on 2 cards ○ 800x600 on 1 card Source: https://en.wikipedia.org/wiki/Scan-Line_Interleave#/media/File:STBVoodoo2SLIcards. jpg

History of Multi-GPU ● 2002 ● ATI Multi-Rendering ● Super Tiling ○ Tiled rendering across N GPUs ○ Where N is “dozens” ● Used by Evans and Sutherland Source: https://hothardware.com/reviews/ati-crossfire-multigpu-technology-preview?page=2

History of Multi-GPU ● 2004 ● Nvidia SLI ○ Now “Scalable Link Interface” ○ Custom PCB, links 2 identical GPUs ● Split-frame Rendering (SFR) ○ Load balanced ● Alternate frame Rendering (AFR) ● DirectX 9 ○ mGPU handled by driver ○ Game-specific profiles Source: https://www.hexus.net/tech/reviews/graphics/916-nvidias-sli-an-i ntroduction/?page=6

History of Multi-GPU ● 2005 ● ATI CrossFire ○ Dual-link DVI Y-dongle ○ Links cards in same family ● Modes: ○ SuperTiling ○ Scissor (SFR) ○ AFR ○ Super AA

History of Multi-GPU ● 2006 ○ Nvidia ■ Quadro Plex up to 8 GPUs ■ GeForce SLI up to 4 GPUs ○ ATI ■ CrossFire -> bridge ■ Bought by AMD ● 2007 ○ CrossFireX up to 4 GPUs

History of Multi-GPU ● 2008 ○ AMD Hybrid CrossFire ■ 780G/V chipset’s Radeon HD3200 integrated GPU ■ Radeon HD3450 discrete GPU ○ Lucid Logix Hydra Engine ● 2009 ○ DirectX 11 ● 2011 ○ AMD Llano APU Dual Graphics ■ SoC IGP + discrete GPU

History of Multi-GPU ● 2013 ○ AMD Mantle ■ Explicit multi-GPU support!! ● 2015 ○ DirectX 12 ● 2018 ○ Vulkan 1.1

Implicit vs Explicit Multi-GPU ● Drivers manage resources ● Engine manages resources ● IHV implements ● Developer implements ● Game only sees 1 GPU ● Game can see all GPUs ● AFR only ● Flexible rendering modes ● Vendor-specific APIs let ● No driver overhead you give the driver hints ● You do the malpractice ● Surgery while wearing yourself!! oven mitts!

Why do Multi-GPU? ● Performance *extremely IHV voice*: to sell more GPUs ● ● Heterogeneous multi-GPU setups common now ● Why not?

Games that use explicit multi-GPU ● Ashes of the Singularity ● Gears of War 4 ● Deus Ex Mankind Divided ● Strange Brigade ● Rise of the Tomb Raider ● Shadow of the Tomb Raider ● Civilization Beyond Earth ● Civilization VI ● Hitman (2016) ● Battlefield 1 ● Sniper Elite 4

Games that use explicit multi-GPU ● Ashes of the Singularity ○ Broad support for 2+ mixed adapters ● Gears of War 4 ○ AFR on 2 linked adapters ● Civilization Beyond Earth ○ SFR

What you can do with Explicit Multi-GPU! ● Hardware Configurations ○ Linked Device Adapters ○ Heterogeneous multi-GPU aka Mixed Device Adapters ● Rendering/Work Distribution Modes ○ Alternate Frame Rendering (AFR) ○ Split Frame Rendering (SFR) ○ Tiled ○ Frame Pipelining ○ Asymmetric

Linked Adapter Multi-GPU ● Pros ● Cons ○ $$$$$$$$$$$$$$$$$ ○ Fast cross-GPU copies ○ Same cards, easy scaling ○ HUGE resolutions e.g. Nvidia Mosaic ■

Linked Adapter Multi-GPU IDXGIAdapter

Heterogeneous Multi-GPU ● Pros ● Cons ○ Can’t assume GPUs ○ Use any GPUs you have! support the same texture layouts. ○ May have vastly different specs/feature support. Compute Units

Heterogeneous Multi-GPU IDXGIAdapter IDXGIAdapter IDXGIAdapter

Alternate Frame Rendering ● Good when you have beefy GPUs and few inter-frame dependencies ● Well-understood ● Issues ○ Need basically the same GPUs for this to work well ○ Frame pacing ○ Input latency ○ Syncing double-buffered stuff ○ Temporal effects

Split Frame Rendering ● Split final frame into even parallel workloads ● Great for VR! ● Good for low input latency ● Load balancing ● Frame compositing ● Not widely used in recent times

Tiled Rendering ● Sorta like SFR but a lot more split up ● Homogenize work all over your entire frame ● Potentially lots of cross-GPU borders ● Even rarer than SFR in recent times

Frame Pipelining ● Copy intermediate steps to the next GPU ● Works better with temporal techniques Source: https://developer.nvidia.com/explicit-multi-gpu-programming-directx-12-part-2

Asymmetric Multi-GPU ● Weak baby integrated GPU and RIPPED DISCRETE GPU? ● As long as you got compute units, you can do Something ● Short trip between iGPU and CPU, save PCIe bandwidth

How do you actually do this ● Enumerate adapters ○ Neat sample that shows both D3D and Vulkan: https://github.com/GPUOpen-LibrariesAndSDKs/VkD3DDeviceMapping ■ ● Find out what features the GPUs support ● Figure out where your resources will live ● Figure out what needs to be synced ○ USE D3DDEBUG/VULKAN VALIDATION ● Figure out what needs to be copied ○ Do copies on the COPY QUEUE!!!!!!!

Challenges to anticipate ● SYNCHRONIZATION ○ Cross-node, cross-adapter, CPU/GPUs, AFR frame sets…..etc……. ● Bandwidth limitations ○ Them texture copies ain’t free ● If you have a Finished™ engine ○ Fixing all the places you assumed you had 1 GPU (heaps, command lists, basically everything) ● Tools? ○ lmao

Tools...? ● GPUView ○ Windows only :’) ● roll your own ● pixel shader printf debugging

GPUView ● Let’s profile!! ● Download here: ○ https://docs.microsofu.com/en-us/windows-hardware/get-start ed/adk-install ○ Part of the Windows Performance Toolkit ● Using D3D12HeterogeneousMultiadapter sample ○ https://github.com/Microsofu/DirectX-Graphics-Samples

GPUView ● [capture walkthrough]

GPUView ● [capture walkthrough]

Conclusion ● It’s cool you should try it!!!! ● Wide open, lots of space for creativity ● BIG CHALLENGE ● If you ever wanted a project to really force you to think about your hardware, here u go

Questions? ● @Icetigris

ADDENDA

Dual GPU Cards ● 1999 ● Quantum3D ○ 2x Voodoo2 SLI board Source: https://en.wikipedia.org/wiki/Scan-Line_Interleave#/media/File:Q uantum3D_Obsidian_X24_SLI_PCI.png

Dual GPU Cards ● 2008 ○ AMD Radeon HD 3850 and 3870 X2

Some PERF NUMBERS ● Gears of War 4 ○ 15.2 ms -> 8.6ms ○ single GeForce GTX 980 Ti -> AFR ● Ashes of the Singularity ○ 17.1 ms ○ Radeon R9 Fury + GeForce GTX 980

GPUView ● [capture walkthrough]

GPUView ● [capture walkthrough]

GPUView ● [capture walkthrough]

GPUView ● [capture walkthrough]