System Considerations in Real Time Video QoE Assessment Amy Csizmar Dalal Department of Computer Science Carleton College adalal@carleton.edu
Outline Architectural overview Design tradeoffs and scenarios Results Conclusions and future work
QoE assessment architecture Goals: Improve system performance, better protocol/network support for Internet video
QoE rating architecture Sampled every second Goal: Examine the data-related design tradeoffs at various points in the rating architecture
Design tradeoffs Tradeoff Description Considerations Range Sampling rate Time between data Missing key congestive 1-5 sec samples events vs. resource utilization Interrating time How much of a False positives/ 10-60 sec stream’s data to negatives vs. missing examine before key congestive events assigning a rating Stream state data How many pieces of “Noisy” data vs. 1, 2, 3, all 4 combos data to use at once, inaccurate data and in what configurations Training set Whether to target the Better chance of a See “scenarios” composition training set to the match vs. resource stream to rate or use utilization all data in the training set Timing concerns Time to train the Flexibility vs. accuracy Fix sample rate system before rating at 1 sec, vary commences interrating time
Scenarios A A A A A Fine-tuned VOD A A A A A A A A A B B B B B General VOD B B B B B B B B B Training set videos General video
Experimental data Name Time Description Action level (MM:SS) cow 1:57 dialog moderate: frequent scene shifts okgo 3:06 music video moderate: stable scene, heavy action up 4:40 animated movie high: frequent scene shifts, short heavy action
Results: Top individual scenarios Scenario Video Stream Sample Time (s) Accuracy state data rate (s) (%) cow TP , BW 2 60 82 Fine-tuned okgo TP , BW 1 20 84 VOD up TP , BW 1 20 80 cow FR 5 50 88 General okgo TP , BW 1 50 86 VOD up TP , BW 1 20 81 cow FR 1 50 83 General okgo TP , BW 2 20 79 video up TP , BW 1 50 75 TP = received packets BW = bandwidth
Results: Top combinations Stream Accuracy (%) Sample Time Scenario state Cow Okgo Up rate (s) (s) Data Fine- Bandwidth + 1 20 77.83 84.10 79.85 tuned received VOD packets General Bandwidth + 1 20 84.73 83.61 80.60 VOD received packets General Bandwidth + 1 20 78.82 78.80 75.19 video received packets
Timing results, general VOD and general video
Conclusions: Tradeoffs summary Tradeoff Best choice Discussion Sampling rate 1 sec Allows maximum detection of congestive events Interrating time 20 sec Stream state Bandwidth + data combos received (Mostly) stream-independent packets Training set All available Fine-tuning does not improve performance composition videos here Training time < 10 minutes Off-line; short enough to allow for worst case retraining flexibility
Timing results, fine-tuned VOD
Recommend
More recommend
