Panoramic video content distribution in the xTV project Peter Quax, - PowerPoint PPT Presentation

Panoramic video content distribution in the xTV project Peter Quax, Panagiotis Issaris, Wouter Vanmontfort, Wim Lamotte Hasselt University, Belgium

Panoramic/omni-directional Video Concept § Comparison : Google streetview with video instead of still images

Panoramic/omni-directional Video Concept § Mobile setups

Panoramic/omni-directional Video Concept § Panoramic super high-definition (22 HD cameras)

Panoramic/Omni-directional video workflow

Processing workflow § Pre-processing delivers a stitched and pre- warped image @ at least 4xFull-HD resolution (or higher if desired/depending on camera) Recordings made at Main Square Rock festival @ Arras (France)

Streaming panoramic video to the end-user

Streaming panoramic video to the end-user What if we could save 80% on bandwidth utilization and still deliver video in its original resolution ?

High-level goals Focus on resolution and quality Ensure low interactivity delays Lower bandwidth and processing requirements Compatibility with existing network technologies and devices

Possible solutions (1) : rescaling Full-HD resolution 4 x Full-HD resolution Compress aggressively Decode entire frame and crop required viewport

Possible solutions (1) : rescaling Full-HD resolution 4 x Full-HD resolution Compress aggressively Decode entire frame and crop required viewport

Possible solutions (2) : transcoding Crop required viewport server-side 4 x Full-HD resolution Decode viewport and display Very fast compression Low quality

Possible solutions (2) : transcoding Crop required viewport server-side 4 x Full-HD resolution Decode viewport and display Very fast compression Low quality

Proposed solution : segmentation

Segmentation – Preparation

Segmentation – Preparation

Segmentation - Encoding

Segmentation - Encoding

Segmentation - Delivery HTTP REQUESTS SEGMENT DELIVERY

Segmentation - Visualization

Segmentation - Visualization

Segmentation - Visualization

Evaluation

Evaluation - compression Parameter Value H.264 Profile Main Preset Medium Tune Fastdecode GOP 16 Segment size 256x216 (unless indicated otherwise) Number of frames 200 Total sequence resolution 3840 x 2160

Evaluation - compression § Storage/compression overhead (on disk) § Versus non-segmented 200 frame sequence @ full resolution § Remark for segmentation approach : this is not what is sent to the client ! (only viewport, baseline has to send everything)

Evaluation - compression § Encoding speed increase § Based on single host (with multiple cores) § Baseline1 uses libx264, multi-threaded § Baseline2 uses libx264 with slice support (real-time optimized)

Evaluation - compression § Encoding scalability § Over multiple hosts § Scaling over hosts not trivial in case of standard codecs (real- time / motion vector limitation)

Evaluation - quality § Remarks on quality comparison § Individual segments need to be encoded using the same quality parameter (VBR, not CBR) to avoid patchwork-effect in tiles § Not all segments compress equally well, total bandwidth required depends on the scene that is being watched § Some examples and terminology in following slides

Evaluation - quality § Least amount of bandwidth (MIN) § Consider case where user is looking at the sky § Minimal motion, best compression

Evaluation - quality § Most bandwidth consumption (MAX) § Consider case where user is looking at the audience § Lots of motion, worst compression

Evaluation - quality § Typical bandwidth usage (AVG) § Consider case where user is looking at main character on stage § Medium motion, medium compression

Evaluation - quality Parameter Value Segment size 256x216 Horizontal segments in viewport 4 Vertical segments in viewport 3 Number of frames 3454 Total sequence resolution 3840 x 2160

Evaluation - quality § Quality comparison (PSNR) § Instruct the baseline codec to use as much bandwidth as the selected viewport (segmentation approach) requires under min/ max/avg conditions (baseline can do CBR !) § Note : the baseline needs to compress/transmit the entire frame (not only the viewport as in segmentation approach)

Evaluation - quality § Quality comparison (SSIM) § Equal conditions as before § Given equal bandwidth allowance, segmentation approach delivers much higher quality

Evaluation – streaming characteristics § Bandwidth utilization under different streaming conditions/container formats § HTTP Live streaming § Uses 10 second fragments

Evaluation – streaming characteristics § MKV over HTTP § Buffers are flushed completely as soon as they are filled

Evaluation – streaming characteristics § MP4 over HTTP § Buffers are gradually flushed and re-filled as required -> TCP flow control

Evaluation – seeking comparison § Rapid seeking is required for segmentation approach § Free/open source codec implementations support seeking to nearest I-frame only (and are often broken in their support) § We need frame-precise seeking ! § Adapt the decoding speed to quickly find the required frame § Support also patched in for HLS (now part of FFMPEG) § Container choice has an impact on seeking performance

Conclusions § Goals achieved ? Yes ! § Bandwidth reduction from 45mbps for full resolution video to 4Mbps § Back-end scalability is ensured through parallelization § Speed of interactivity optimized through pre-caching and enhanced seeking methods § Solution is fully compliant with existing distribution methods and current-generation tablet hardware § Only ideas/theory ? No ! § We have a fully working prototype based on a second screen setup (STB and tablet) § User experience is under investigation, early feedback is very positive

Questions ? peter.quax@uhasselt.be