Introduction (1 of 2) An Empirical Evaluation of VoIP Playout Buffer - PDF document

4/3/2015 Introduction (1 of 2) An Empirical Evaluation of VoIP Playout Buffer Dimensioning in • VoIP increasingly important Skype, Google Talk, and MSN – Started with inexpensive use at home with friends and family Messenger – Now businesses between corporations • Sound quality can be comparable to S d lit b bl t Chen ‐ Chi Wu, Kuan ‐ Ta Chen, Yu ‐ Chun traditional telephones Chang, and Chin ‐ Laung Lei • Skype reports: 405 million registered users, 15 ACM Workshop on Network and Operating million online users [footnote 1] System Support for Digital Audio and Video • Reliable service and quality a priority for ISP (NOSSDAV) Williamsburg, VA, USA and VoIP providers June 2009 Introduction (2 of 2) Buffering Basics • Many factors impacting quality • Sacrifice speech conversational interactivity for better sounding quality playout • (This class talks about a lot of them!) – “Smoother” sound, plus could repair loss – Codec, Transport protocol, Redundancy and Error • Typically, transmit packets every 30 ms, but can Typically, transmit packets every 30 ms, but can Control and Playout Buffer Control, and Playout Buffer arrive later than 30 ms from previous (delay jiiter) • This work focuses on the Playout Buffer – Results is silent periods, noise, unclear speech (depending upon loss concealment) • So, playout buffer holds packet temporarily in order to allow more packets to arrive on time 1

4/3/2015 Buffering Challenge Buffering in Practice • How to determine best playout buffer size to • Academics proposed many algorithms [9 ‐ 11, 13] use? • Most adjust buffer based on linear combination of • Larger buffer leads to better sounding voice network delay and jitter quality, but lower interactivity and vice versa q y, y – Combinations vary with network measurements • But what algorithms are used in practice? • Optimal size affected by network delay, delay jitter, repair and compression (codec) • Analyze 3 popular VoIP applications: Skype , Google implementations Talk , MSN Messenger – And network factors may change over time, so – Do they differ? buffer size should too! – Do they adjust? – How close to “optimal”? Outline Related Work (1 of 2) • [11]: Authors use weighted exponential moving average of • Introduction delay and standard deviation to determine buffer • Related Work – weights are hard ‐ coded • Experiments • [10]: extends [11] by adapting the weights according to magnitude of events magnitude of events • Results – Both [10] and [11] by simulation • Optimal • [9,13]: extend by adjusting during talk spurt so can adapt to changes in network more quickly • Conclusion • Above, all academic systems  What is used in practice? 2

4/3/2015 Related Work (2 of 2) Outline • To assess, Perceptual Evaluation of Speech Quality • Introduction (PESQ) [8] • Related Work – Compare original to degraded, and map to Mean Opinion Score (MOS), value 1 ‐ 5. • Experiments • E ‐ Model has arithmetic sum of impairments of delay E Model has arithmetic sum of impairments of delay, • Results equipment and compression [7] • Optimal – R = 94 – i (delay) – i (loss)  R factor , can map to MOS • Neither is sufficient. PESQ does not use delay, E ‐ • Conclusion model not accurate nor combines delay and quality • [5] combines both  Use their technique (later) Experiment Methodology Buffer Size Estimation Free BSD w/dummynet as router • • Have two audio samples. Compare to – Control loss, delay, jitter (stddev determine delay (use cross ‐ correlation of delay) voice – Link is 1 Mb/s coefficient [1]) 2 PCs running Windows XP with • Skype, Google Talk, MSN Messenger – (MLC: not validated as a technique?) – One PC talker the other One PC “talker” the other • Note, not sure of sample interval, “listener” Play recording on talker, send to • compression, etc. (“black box”) listener degraded – Recording from Open Speech – But, estimate to be 50 msec based on literature voice Repository [3] Record both talker and listener • May not be totally accurate, but want to see • Each “call” 240 seconds • speech 10 calls at each setting how commercial VoIP applications adjust • – Compare to get degradation 3

4/3/2015 Outline Network Delay and Jitter • Introduction • Related Work • Experiments • Results • Optimal • Delay: • Jitter: • Conclusion – Skype doesn’t adjust – Skype flat, so doesn’t adjust – MSN doesn’t adjust – Google adjusts slightly, lots of variance – Google may (fig b, trendlines differ). – MSN adjusts linearly Network Loss Rate Outline • Introduction • Related Work • Experiments • Results • Optimal • Conclusion • All flat, so no apparent adaptation 4

4/3/2015 QoE Measurement Model Determining Optimal Buffer Size • Based on [5] … • Yields best quality (QoE, previous slide) • Encode audio clips from open speech repository [3] • Given original and degraded clips to VoIP using [2] • Apply PESQ to get MOS – Use G.711, popular codec • Convert MOS to R score • Simulate any loss (using Gilbert model) • Simulate any loss (using Gilbert model) – Using formula in ITU ‐ T G.107 [7] i f l i [ ] • Compute delay impairment (I d ) from E ‐ model • Add delay (Gamma distribution) I d = 0.024 x d if d < 177.3 – If later than buffer size, drop • (MLC: what policy is this?) I d = 0.024 x d x (d – 177.3) if d > 177.3 • Decode any resulting stream • Subtract I d from R score to get R’ • Apply QoE to determine quality • Convert R back to MOS Optimal Buffer Size with Loss Optimal Buffer Size with Delay and Jitter • Method: Delay all 100 ms, add loss • Method Delay all 100 ms add loss • As expected, MOS decreases with delay (sanity check) • MOS varies a lot with buffer size • Loss degrades MOS (sanity check) – Important to get buffer size right • With jitter, optimal point shifts left with • Optimal indicated by ‘X’ higher loss – As jitter increases, more delay is necessary – May be different with repair (future work) 5

4/3/2015 Optimal for Skype, Google, MSN Model for Determining Optimal Buffer Size • Can derive optimal via simulations – But lot of work, not real ‐ time • Try regression to determine under network scenario Optimal buffer = • Delay – average network delay, jitter – std of delay, plr ‐ packet loss rate • For G.711, coefficients are below, R 2 is 0.885 (good) • (They don’t adjust for loss, so no further analysis) • All are conservative (~220 ms buffer) with no jitter • MSN adapts best with jitter, others too conservative Conclusions Future Work? • Investigate if gap between academic research and practice exists – MSN Messenger, Skype, Google Talk • MSN best in terms of buffer dimensioning • MSN best in terms of buffer dimensioning • Skype, does not adjust much at all • Provide algorithm to compute optimal based on QoE metric and model 6

4/3/2015 Future Work • More factors – Frame size – Repair – Codec Codec • Use optimal dimensioning model in system – Real ‐ life experiments to evaluate 7