PARSEC: Streaming 360 o Videos Using Super-Resolution Mallesham - PowerPoint PPT Presentation

PARSEC: Streaming 360 o Videos Using Super-Resolution Mallesham Dasari, Arani Bhattacharya, Santiago Vargas, Pranjal Sahu, Aruna Balasubramanian, Samir R. Das Department of Computer Science https://www3.cs.stonybrook.edu/~mdasari/parsec

360 o Vi 360 Video Streaming q Central to many immersive applications (e.g., VR/AR) Image credit: Oculus Immersive Experience $ Billion Market Popularity of 360 o Video is on the Rise! http://blog.dsky.co/tag/head-tracking/

Gr Gran and Challe allenge q 360 o videos require 8x bandwidth compared to regular videos for the same perceived quality 360 o 110 o 25Mbps 80 o 180 o Image from Rollercoaster video 200Mbps http://blog.dsky.co/tag/head-tracking/

Cu Curr rrent Sol Solution ons q Viewport-adaptive streaming • Divide video into tiles (e.g., 192x192 pixels) Flare [MobiCom’18], Rubiks [MobiSys’18], MOSAIC [IFIP Networking’19] PANO [SIGCOMM’19], ClusTile [INFOCOM’19] 4

Cu Curr rrent Sol Solution ons q Viewport-adaptive streaming • Divide video into tiles (e.g., 192x192 pixels) • Predict viewport tiles based on head tracking and video saliency analysis • Stream only viewport specific tiles using ABR algorithm Flare [MobiCom’18], Rubiks [MobiSys’18], MOSAIC [IFIP Networking’19] PANO [SIGCOMM’19], ClusTile [INFOCOM’19] 5

Li Limi mitation ons of of Cu Curr rrent Sol Solution ons q Viewport Prediction (VP) Mosaic 100 [IFIP Networking’19] Accuracy (%) • Predicting user head movement is 80 hard 60 • Fetch more tiles to avoid the tile 40 misses Flare 20 • Fetching more tiles competes for [MobiCom’18] bandwidth and reduces video 1 2 3 quality Prediction Window (seconds) q Network is the only resource for achieving good video quality Can we improve client’s video quality without relying much on network? 6

Op Opportunity1 y1: Super-re resolution q Use low resolution image/video, hallucinate the details to produce high resolution https://amundtveit.com/2017/06/04/deep-learning-for-image-super-resolution-scale-up/ • Idea dates to the 90s • Currently benefiting from deep neural networks (DNNs) DNNs are computationally expensive 7

Op Opportunity2 y2: Computation q Significant improvement in GPU capacity over the decade • Often underutilized q Leverage this compute capacity on the client to do super- resolution NAS [OSDI’2018] Is this compute power enough to do super-resolution? 8

Su Super-re resolution Challenges q Bulky DNN models • Slower inference (e.g., less than 2FPS for a 1- minute 4k video) • Large model sizes Model trained for one-minute video duration q Large variance in quality enhancement How to make the models smaller, faster & better? 9

Li Lightweight Mi Micr cro-mod models fo for Su Super-re resolution q Train a model for each segment 360 192x192 192x192 24x24 24x24 Video Segment . . . . . . . . 3840x1920 . . . . (20x10 Tiling) Reconstructed Original Down- Compressed High-Quality High-Quality Sampled ULR Tiles Tiles Tiles Tiles q Fetch the model along with segment download q Enhance the quality of few viewport-specific tiles instead of whole frame 10

Lightweight Mi Li Micr cro-mod models fo for Su Super-re resolution q Benefits q Additional challenges ü Small model footprint • Still only few tile/sec inference rate ü Faster inference q Key Questions • Which tiles to download and at what quality? • Which tiles to generate (using super-resolution)? • Which tiles to ignore? Need a new ABR algorithm that combines compute and network resources 11

Ne Neural-Aw Aware ABR Compute GPU (SR model) Capacity Quality3 Quality2 Quality1 ULR tiles Network Request tiles Capacity Respond tiles Client Player Player Buffer Video Server 0.09 0.09 0.43 0.43 0.31 0.31 0.19 0.19 0.21 0.21 0.02 0.02 0.09 0.09 0.24 0.24 0.98 0.98 0.98 0.98 0.63 0.63 0.08 0.08 0.14 0.14 0.21 0.21 0.99 0.99 0.96 0.96 0.56 0.56 0.11 0.11 0.02 0.02 0.13 0.13 0.23 0.23 0.34 0.34 0.27 0.27 0.12 0.12 Viewport Prediction 12 [IFIP Networking’19]

Ne Neural-Aw Aware ABR How to Find a Solution Fast? Greedy Algorithm Maximize 13

Pu Putting Eve verything To Together Client HTTP Server Offline Viewport Super Prediction Neural-Aware ABR Algorithm ULR Tiles Resolution & Micro- Model Models Network Inference Training Bitrate State Scheduler Selection for Compute for Downloaded Capacity Generated HEVC Only Tiles Tiles Encoded these Segments Generated and components Decoded Playback Buffer Downloaded Tile Render and Display Qualities present in state-of-the- art 360° video streaming 14

Im Implem plemen entatio tion n & & Evalua aluatio tion • Linux server • Diverse network conditions • Node.js • Real traces: WiFi & • Client 4G/LTE • Pixel3 phone • FCC & Belgium traces • Super-resolution model • 360 o video dataset • Keras with Tensorflow • 10 videos backend • MMSYS’17 head movement dataset 15

Pe Performance Comparison • VP_Only [NOSSDAV’17] • NAS-regular [OSDI’18] • Download only • A recent regular video viewport-specific tiles streaming system using super-resolution • FLARE [MobiCom’18] • NAS-360 • Fetch additional tiles to accommodate VP • A modified version of inaccuracy NAS-regular for 360 o video 16

Pe Performance Comp Compari rison on Average Quality y and Tile Misses 26% improvement at 30% improvement the 90 th percentile compared to Flare compared to Flare [MobiCom’18] [MobiCom’18] 17

Ov Overall QoE QoE Pe Performance 37% improvement compared to Flare [MobiCom’18] 18

Im Impac pact t of Comput putatio tion PARSEC performs better as we increase the computing power 19

Con Conclusion on • PARSEC • A panoramic video streaming system • DNN based super-resolution • Neural-aware ABR algorithm • PARSEC provides high QoE compared to the state- of-the-art solutions For more details please visit: https://www3.cs.stonybrook.edu/~mdasari/parsec 20