1/13/2018 Spherical Panoramic Images LightDB A Database System for Virtual, Augmented, & Mixed Reality Video Applications Brandon Haynes, Amrita Mazumdar, Armin Alaghi, Magdalena Balazinska, Luis Ceze, & Alvin Cheung 2 Spherical Panoramic (360⁰) Videos Light Fields What happens here? 3 4 LightDB LightDB Key Features: Today: • Data management system for VR/AR/MR video applications 1. LightDB Data Model • Unified data model for panoramic (360 ° ) and light field video • Declarative queries with automatic optimization 2. Physical & Logical Algebra • Full stack: data ingest, processing, and real-time streaming 3. Architecture & Optimizer Key Results: 4. Application: Predictive 360 ° Streaming • Decreased development complexity ( ∼ Τ 1 10 LOC) • Increased performance (up to 4 × for real-world workloads) • Reduced client bandwidth & power requirements 5 6 1
1/13/2018 Spherical Images 360⁰ Videos Light Fields The Light Field Data Model ! Angle ( 𝜄, 𝜚 ) Angle ( 𝜄, 𝜚 ), Time ( 𝑢 ) Angle ( 𝜄, 𝜚 ), Time ( 𝑢 ), Position ( 𝑦, 𝑧, 𝑨 ) 7 8 Light field data model 𝐺 0, 0, 0, 𝑢 1 , 𝜄, 𝜚 = red Light field data model 𝐺 0, 0, 0, 0, 𝜄, 𝜚 = red 𝑧 𝑧 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 𝑨 𝑨 𝜄 𝜄 Panoramic Image 𝜚 𝜚 𝑦 𝑦 Time 𝑢 Time 𝑢 9 10 Light field data model 𝐺 0, 0, 0, 0, 𝜄, 𝜚 = red Light field data model 𝐺 0, 0, 0, 0, 𝜄, 𝜚 = red 𝑧 𝑧 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 𝑨 𝑨 𝜄 𝜄 Panoramic Image 360⁰ Videos 𝜚 𝜚 𝑦 𝑦 Time 𝑢 Time 𝑢 𝐺 0, 0, 0, 𝑢, 𝜄′, 𝜚′ = orange 𝐺 0, 0, 0, 0, 𝜄′, 𝜚′ = green 11 𝐺 0, 0, 0, 0, 𝜄′, 𝜚′ = green 12 2
1/13/2018 LightDB Model Light field data model 𝐺 0, 0, 0, 0, 𝜄, 𝜚 = red 𝑧 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 … LightDB Algebra User Query Query SELECT 𝜄 𝑨 𝐺 𝒚, 𝑧, 𝑨, 𝑢′, 𝜄 ′′ , 𝜚′′ = blue PARTITION UNION Light Fields MAP Light Fields 𝐺 𝑦, 𝑧, 𝑨, 𝑢, 𝜄, 𝜚 𝜚 𝑦 SUBQUERY ROTATE Time 𝑢 TRANSLATE 2D Videos, ENCODE Geometric Models, Filesystem … 𝐺 0, 0, 0, 𝑢, 𝜄′, 𝜚′ = orange DECODE 𝐺 0, 0, 0, 0, 𝜄′, 𝜚′ = green 13 14 LightDB Algebra LightDB Query Language ′ , 𝑧 1 ,𝑧 1 ′ , 𝑨 1 ,𝑨 1 ′ , 𝑦 1 ,𝑦 1 𝑠 2 , , 𝑐 PARTITION (Δ𝑧) [−∞, +∞], 𝑠, , 𝑐 ↦ SELECT MAP 2 0, 2𝜌 , 0, 𝜌 Decode( INPUT . MP 4 ) >> Union(Scan(watermark)) >> Map( GRAYSCALE ) >> Encode( HEVC ); Actual C++ statement! 15 16 Decode( INPUT . MP 4 ) Decode( INPUT . MP 4 ) >> Union(Scan(watermark)) >> Union(Scan(watermark)) >> Map( GRAYSCALE ) NVENC >> Map( GRAYSCALE ) Encode Encode >> Encode( HEVC ) >> Encode( HEVC ) HEVC Encode HEVC CUDA Map Map (Grayscale) (Grayscale) Kernel CUDA Union Union Kernel NVDEC NVDEC Decode Scan Decode Scan Decode Decode input.mp4 watermark/metadata.mp4 Logical Plan Logical Plan Physical Plan 17 18 3
1/13/2018 Watermark Query Performance Decode( INPUT . MP 4 ) >> Union(Scan(watermark)) NVENC Encode >> Map( GRAYSCALE ) 60 >> Encode( HEVC ) Encode HEVC Composed Map (Grayscale) CUDA Kernel 40 LightDB Architecture FPS NVDEC NVDEC Union Decode Decode 20 GOP-GPU GPU Decode Scan Shuffle Broadcast 0 input.mp4 watermark.mp4 Optimized Physical Plan Logical Plan 19 20 LightDB Architecture LightDB Architecture User Queries Query Processor User Queries Query Processor Physical Operators Physical Operators GPU CPU FPGA GPU CPU FPGA Homomorphic Homomorphic Operators Operators Operators Operators Operators Operators Operators Operators Input Camera Rigs VR Viewers Input Camera Rigs VR Viewers Edge Edge Preprocess Plan Execute Preprocess Plan Execute Cache Cache VR Processing Hardware Client Hardware VR Processing Hardware Client Hardware Catalog Optimizer Catalog Optimizer • (Currently) Rule-based • Remove redundant operations Pro rocessing at Pro rocessing at • Pro rocessing at Storage Manager Pro rocessing at Storage Manager Push down interpolation the Edge the Edge • the Edge the Edge Avoid inter-device transfer • In-Memory Persistent In-Memory Persistent Transform physical representation Buffer Storage Buffer Storage 21 22 LightDB Architecture GPU-Based Map Operator @ 4K 30 User Queries Query Processor Physical Operators GPU CPU FPGA Homomorphic FPS Operate on physical video Operators Operators Operators Operators representation WITHOUT an (expensive) decode Input Camera Rigs VR Viewers Edge Preprocess Plan Execute Cache VR Processing Hardware Client Hardware 0.1 0 Catalog Optimizer Pro rocessing at Pro rocessing at Storage Manager Gaussian Blur the Edge the Edge In-Memory Persistent Buffer Storage LightDB FFmpeg OpenCV SciDB 23 24 4
1/13/2018 Homomorphic Tile Union Operator @ 4K 100,000 10,000 LightDB Application: 1,000 FPS Predictive Panoramic Tiling 100 10 Preconditions: • HEVC Codec with Tiles 0.1 • Non-Overlapping Union 1 • Temporal Alignment LightDB FFmpeg OpenCV SciDB 25 26 encoding.com encoding.com 27 28 Predicted Orientation Bitrate = 2000 kbps 29 30 5
1/13/2018 Bitrate = 50 kbps Predicted Predicted Adjacent Tile Adjacent Tile Adjacent Tile Adjacent Tile Orientation Orientation Bitrate = 500 kbps Bitrate = 500 kbps Bitrate = 500 kbps Bitrate = 500 kbps Bitrate = 2000 kbps Bitrate = 2000 kbps No one looks at the ground! Bitrate = 50 kbps 31 32 LightDB Queries Current VR Video Applications $ ffmpeg UNION UNION -fpsprobesize 0 -i input2.h264 -g 30 -r 30 -filter_complex "[0:v]crop=384:320:0:0[tile0]; [0:v]crop=384:320:384:0[tile1]" … … -map [tile0] -g 30 -r 30 Predictions ENCODE ENCODE ENCODE ENCODE -preset slow -qmin 10 -qmax 50 1 1 𝑟 1 1 , …, 𝑟 𝑜 𝑜 ⋅, HEVC 𝑟 1 ⋅, HEVC 𝑟 𝑜 ⋅, HEVC 𝑟 1 𝑜 ⋅, HEVC 𝑟 𝑜 𝑜 -b:v $TILE0_BITRATE Decode(rtp ://…) -f segment STORE -segment_format mp4 >> Partition(Time, 1) -segment_format_options (⋅, OUTPUT ) movflags=+faststart+frag_keyframe+default_base_moof >> Partition(Theta, 90⁰) -segment_time 2 … tile-0-\%d.mp4 >> Partition(Phi, 45⁰) -map [tile1] -g 30 -r 30 PARTITION PARTITION ENCODE >> Transcode( 𝑔 ) -preset slow -qmin 10 -qmax 50 (⋅,𝑢, Δ𝑢) (⋅,𝑢, Δ𝑢) (⋅, HEVC ) -b:v $TILE1_BITRATE >> Store(output); -f segment -segment_format mp4 -segment_format_options movflags=+faststart+frag_keyframe+default_base_moof PARTITION -segment_time 2 UNION tile-1-\%d.mp4 (⋅, Δ𝜄, Δ𝜚) $ ffmpeg ! -r 60 -i tile-0-0.mp4 -i tile-1-0.mp4 -i tile-0-1.mp4 -i tile-1-1.mp4 D ECODE -i tile-0-2.mp4 -i tile-1-2.mp4 -i tile-0-3.mp4 -i tile-1-3.mp4 ⋅, H 264 -i tile-0-4.mp4 -i tile-1-4.mp4 -filter_complex "[0:v][1:v]vstack[out0]; [2:v][3:v]vstack[out1]; [4:v][5:v]vstack[out2]; [6:v][7:v]vstack[out3]; [8:v][9:v]vstack[out4]; [out0][out1][out2][out3][out4]concat=n=5[out]" -map [out] Logical Plan output.hevc 33 34 HOMOMORPHIC TILE … Predictions GPU - ENCODE 𝑟 1 GPU - ENCODE 𝑟 12 1 , …, 𝑟 12 𝑜 𝑟 1 STORE GPU - PARTITION 4,3 (⋅, OUTPUT ) HOMOMORPHIC GPU - DECODE CONCATENATE LightDB HOMOMORPHIC PARTITION Physical Plan 6
1/13/2018 Predictive Panoramic Tiling @ 4K LightDB Brandon Haynes 60 bhaynes@cs.washington.edu Key Features: FPS • Data management system for VR/AR/MR video applications 30 • Unified data model for panoramic (360 ° ) and light field video • Declarative queries with automatic optimization • Full stack: data ingest, processing, and real-time streaming 0 Key Results: 4K • Decreased development complexity ( ∼ Τ 1 10 LOC) • Increased performance (up to 4 × for real-world workloads) Resolution • Reduced client bandwidth & power requirements LightDB FFmpeg OpenCV SciDB 37 38 7
Recommend
More recommend
