Mobile Video Quality Assessment Database Lark Kwon Choi with Anush - PowerPoint PPT Presentation

Mobile Video Quality Assessment Database Lark Kwon Choi with Anush K. Moorthy, Prof. Alan C. Bovik, and Prof. Gustavo de Veciana

Outline Introduction LIVE Mobile VQA Database Subjective Study Source videos and distortion simulation Test methodology Evaluation of Subjective Opinion Discussion and Conclusion 2

Growth in Mobile Video Traffic More devices, higher bit rates contents 78 % Global Mobile Data Traffic Growth by 2016 Mobile Video Traffic 70.5 % How? Need for more video capacity, viewer’s QoE 3

Promising Direction “Perceptual optimization” of video networks Video Compression Transmission Visual Perception Wireless Networks Feedback Humans are final “receivers” of videos To understand human’s opinion and behavior on visual quality, HD Mobile VQA Database & Subjective Analysis 4

Previous Subjective Studies Focus Limits Subjective studies [K. Seshadrinathan et al., 2010] • Large displays • Results cannot be [A.K. Moorthy et al., 2010] Distortion: translated into small • [VQEG VQA Phase I and II, 2000, 2003] Compression, mobile devices [VQEG Multimedia Phase I, 2008] IP/wireless loss [S.R. Gulliver et al., 2007] Delayed and • [Q. Huynh ‐ Thu et al., 2006] jitter [A. Eichhorn et al., 2009] • Mobile devices • Small datasets [H. Knoche et al., 2005] Insufficient distortions • [S. Jumisko ‐ Pyykko et al., 2005, 2008] • Unknown source [M. Ries et al., 2007] Small resolution • [S. Winkler et al., 2003] Lack of publicity • To aid the development of perceptually optimized algorithms for wireless video transmission, LIVE Mobile VQA Database 5

10 Reference, 200 distorted videos, and >50 subjects

Source Videos Digital Cinematographic Camera 12bit REDCODE RAW 2K (2048 x 1152), 30/60 fps RED ONE 10 Actual Study 2 Training Downsampled 720p (1280 x 720), Uncompressed YUV, 15 sec, Reference videos 7

Distortion Simulations H.264 Compression Wireless Packet Loss Frame freezes Rate adaptation Temporal dynamics 8

Distortions 4 Compression + 4 Wireless Packet Loss ‐ JM H.264 SVC / R 1 < R 2 < R 3 < R 4 / 0.7 ~ 6M using fixed QP encoding quality IEEE 802.11 Q 4 Wireless Q 3 channel Q 2 Simulator ‐‐‐‐‐‐‐‐‐‐‐‐‐‐ Q 1 Perceptual QAM separation OFDM of video quality modulation R 4 source rate R 1 R 2 R 3 SVC layers 9

Time Varying Distortions 4 Frame freezes 1sec short stored video freezing 4sec long stored video freezing 4sec long LIVE video freezing 3 Rate adaptation Layers (R1 ~ R4) Time (15sec) 5 Temporal dynamics 10

Test Methodology Single ‐ stimulus continuous quality evaluation (SSCQE) with hidden ‐ reference Phone (Motorola Atrix) ‐ 200 videos, 36 subjects, 18 ratings, 2 rejects Real ‐ time quality evaluation End ‐ of ‐ video quality evaluation 11

Result of Subjective Study DMOS scores and corresponding histogram We assume that the DMOS scores have a Gaussian distribution 12

Evaluation of Subjective Opinion Statistical analysis of human behavior

Compression Conducted t ‐ test by using DMOS values ‐ 95 % confidence level ‐ compared in separate contents Higher rate is better R1 R2 R3 R4 R1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R2 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 R4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ • ‘ 1 ’ : row algorithm is ’statistically better’ than the column algorithm • ‘ 0 ’ : ‘ worse ’ and ‘ ‐ ’ : ‘ identical ’ • Each entry of the matrix represent 10 reference videos 14

Frame Freeze 1sec short stored video freezing 4sec long stored video freezing 4sec long LIVE video freezing Choppy freezing is worse Frame lost is worse Freeze – 1sec Freeze – 2sec Freeze – 4sec Real ‐ time Freeze 4 sec Freeze – 1sec ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 ‐ 0 0 0 ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Freeze – 2sec 1 1 1 ‐ 1 1 1 ‐ 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 ‐ 1 ‐ 1 0 ‐ 0 Freeze – 4sec 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1 Real ‐ time Freeze 4 sec 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ 0 ‐ 0 1 ‐ 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 15

Rate Adaptation Higher rate is better R1 ‐ R4 ‐ R1 R2 ‐ R4 ‐ R2 R3 ‐ R4 ‐ R3 R1 ‐ R4 ‐ R1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R2 ‐ R4 ‐ R2 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 R3 ‐ R4 ‐ R3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ Average stable constant rate is better R1 R2 R3 R4 R1 ‐ R4 ‐ R1 1 1 1 1 1 1 1 1 1 1 0 0 0 ‐ 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R2 ‐ R4 ‐ R2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R3 ‐ R4 ‐ R3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ 0 1 ‐ 0 1 0 0 0 0 0 0 0 0 0 0 0 16

Temporal Dynamics Multiple rate switches are better R1 ‐ R4 ‐ R1 R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1 R1 ‐ R4 ‐ R1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 ‐ ‐ ‐ 0 0 0 0 1 ‐ R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1 1 ‐ ‐ ‐ 1 1 1 1 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ Ending on a higher quality is preferable R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1 R1 ‐ R2 ‐ R4 R4 ‐ R2 ‐ R1 R1 ‐ R3 ‐ R4 R4 ‐ R3 ‐ R1 R1 ‐ R4 ‐ R1 ‐ R4 ‐ R1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 1 1 0 0 0 0 1 1 ‐ 1 1 1 1 1 1 1 0 0 0 0 ‐ 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1 R1 ‐ R2 ‐ R4 ‐ 1 1 1 0 0 1 1 1 1 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1 0 0 ‐ 0 0 0 ‐ 0 0 0 1 1 1 ‐ 1 1 1 1 1 1 R4 ‐ R2 ‐ R1 0 0 ‐ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 0 0 0 0 0 0 0 0 0 0 1 ‐ 0 0 0 0 ‐ 0 1 0 R1 ‐ R3 ‐ R4 1 1 1 1 1 1 1 1 1 1 1 1 1 ‐ 0 ‐ 0 1 ‐ 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 1 1 1 1 1 1 1 1 1 1 R4 ‐ R3 ‐ R1 0 0 1 1 0 0 0 0 0 0 0 0 ‐ 0 0 0 0 0 0 0 ‐ 1 1 1 1 ‐ 1 0 1 0 0 0 0 0 0 0 0 0 0 ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ 17

Discussion and Conclusion LIVE Mobile VQA Database 10 reference, 200 distorted videos, and Over 50 subjects 4 compression + 4 wireless packet loss + 4 frame freezes + 3 rate adaptation + 5 temporal dynamics Analysis of human behavior Higher, stable bit rates, and multiple efforts Choppy freezing, frame lost Getting better (ending) quality Invite further analysis of human behavior 18

Questions ? Acknowledgment ‐ NSF CCF ‐ 0728748 ‐ Intel & Cisco VAWN program For more explanations, A.K. Moorthy, L ‐ K. Choi, A.C. Bovik, and G. de Veciana, “Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies,” IEEE Journal of Selected Topic in Signal Processing. Special Issue on New Subjective and Objective Methodologies for Audio and Visual Signal Processing, 2011. (re ‐ submitted after revision) 19