Level elUp: A Thin-cl cloud Appr pproac ach h to Game e Lives - PowerPoint PPT Presentation

Level elUp: A Thin-cl cloud Appr pproac ach h to Game e Lives estrea eaming Landon Cox (Microsoft Research) Lixiang Ao (UC San Diego)

Game Livestreaming • Twitch • Average >2m concurrent viewers and >90k concurrent channels • >65m hours streamed and >1.5b hours watched per month • Market size $40b, expected to grow 18% per year

Game Livestreaming

Cloud-based Video Transcoding Is Expensive • $300+ to transcode 100 hours of video on Azure • <$20 to livestream 100 hours of single-bitrate video on Wowza • Reason: Video transcoding is resource demanding, usually requires hardware accelerators in the cloud

Mobile Hardware Trends GPU video codec ML

Mobile Hardware Trends • As of May 2018, hundreds of millions devices equipped with powerful GPU/hardware codec/ML accelerators • Most smartphones are expected to have ML accelerators in few years • Combined capabilities of hardware accelerators on mobile devices are greater than the cloud • The edge is ready to play a more central role in video livestreaming

LevelUp: A Thin-cloud Approach to Game Livestreaming • Offload cloud-based transcoding by encoding multi-bitrate videos on broadcasters’ smartphones • In case of bandwidth constraints, viewers boost reduced-resolution video quality with super-resolution using ML • Adopt game-specific CNN models to improve quality

LevelUp Design What if bandwidth is not enough?

LevelUp Design – bandwidth constrained +

Single-image super resolution (SR) • CNNs upscale resolution of reduced-resolution images • Mobile ML accelerators enable fast NN computation • A lightweight CNN model of 4 layers is sufficient for LevelUp • Different games have visual features, require different models • Models are trained offline, downloaded to viewers before streaming

Broadcaster’s video pipeline Capture screen Upload to server Resize to 3 resolutions Encode segments Send to HW encoder queue

Viewer’s video pipeline – without SR Decode Download video segment Display

Viewer’s video pipeline – with SR SR Decode Bicubic Download video segment interpolate Merge, display Separate grayscale, chroma

Evaluation • Can super resolution improve the visual quality of reduced-resolution game streams? • Can broadcasters perform multi-bitrate encoding in realtime? • Can viewers super-resolve reduced-resolution video streams in realtime? • What is LevelUp’s energy overhead?

Game stream super-resolution PSNR: per-pixel, SSIM: structural, # of improvements with SR VMAF[1]: ML model of human perception PSNR SSIM VMAF PSNR SSIM VMAF 100% 100% 90% 90% 80% 80% Percent change Percent change 70% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% 0% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft Witcher 3 CSGO Dota 2 Fallout 4 GTAV Rust Starcraft Witcher 3 2 2 480p + SR 270p + SR [1] https://github.com/Netflix/vmaf

Game stream super-resolution VMAF improved 61-88% VMAF improved 15-20% PSNR SSIM VMAF PSNR SSIM VMAF 100% 100% 90% 90% 80% 80% Percent change Percent change 70% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% 0% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft Witcher 3 CSGO Dota 2 Fallout 4 GTAV Rust Starcraft Witcher 3 2 2 480p + SR 270p + SR [1] https://github.com/Netflix/vmaf

Game stream super-resolution Interpolated, 480x270 LevelUp, 480x270

Game stream super-resolution Interpolated, 480x270 LevelUp, 480x270

Quality vs. Bitrate GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR) 100 90 80 70 60 VMAF 50 40 30 20 10 0 0.1 1 10 100 Bitrate (Mbs)

Quality vs. Bitrate GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR) 100 90 80 70 60 VMAF boost by LevelUp VMAF boost by LevelUp VMAF 50 40 30 20 10 0 0.1 1 10 100 Bitrate (Mbs)

Quality vs. Bitrate GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR) 100 90 80 Medium+SR vs. High Bitrate: 90% lower bandwidth, 10-20% VMAF loss 70 60 VMAF 50 Low+SR vs. Medium Bitrate: 40 70% lower bandwidth, 3-20% VMAF loss 30 20 10 0 0.1 1 10 100 Bitrate (Mbs) Super-resolution can greatly improve game stream quality w/o extra bandwidth consumption

Can broadcaster encode multi-bitrate streams? • Encode 3 2-second segments (1080p, 480p, 270p) at the same time • Test on iPhone 11 Pro, the iPhone Xs, iPhone 8, iPhone 7, and iPhone 6s • All devices can encode in realtime • Multi-bitrate encoding is feasible on modern smartphones

Can viewers super-resolve video? 0.2 0.18 0.16 0.14 0.12 Seconds 0.1 0.08 0.06 0.04 30FPS-capable 0.02 0 iPhone 11 iPhone Xs iPhone 8 iPhone 7 iPhone 6s (A13) (A12) (A11) (A10) (A9) Recent smartphones equipped with ML accelerators can super-resolve gaming streams in realtime.

Energy overhead 100 } -5% after 30 min streaming 90 80 % charge remaining 70 60 50 40 30 20 10 0 Low Med Low-SR Low-SR (iPhone Xs) (iPhone Xs) (iPhone Xs) (iPhone 11 Pro) LevelUp has small energy overheads even with super-resolution enabled.

Conclusion • Game livestreaming is expensive due to realtime transcoding • LevelUp can greatly reduces game livestreaming costs by leveraging smartphones for transcoding • LevelUp uses super-resolution to boost quality for reduced-resolution videos by up to 88% • LevelUp can transcode and super-resolve game streams in realtime using recent smartphone hardware accelerators

Thank you! Q & A Landon Cox (Microsoft Research) landon.cox@microsoft.com Lixiang Ao (UC San Diego) liao@ucsd.edu