Its Not the Cost, Its the Quality! Ion Stoica Conviva Networks and - PowerPoint PPT Presentation

It’s Not the Cost, It’s the Quality! Ion Stoica Conviva Networks and UC Berkeley 1

A Brief History � Fall, 2006: Started Conviva with Hui Zhang (CMU) � Initial goal: use p2p technologies to reduce distribution costs and improve the scale � Slowly, realized our customers (content premium producers & aggregators) value more quality than cost � Today: maximize distribution quality, distribution management, and provide real-time analytics 2

Where is the Data Coming From? � Content Providers and Aggregators � CDNs 3

Trends 4

Trends: CDN Pricing � CDN pricing has decreased x1.5-2 every year over the last 5 year 45 40 35 30 cents/GB 25 20 15 10 5 0 2006 2007 2008 2009 2010 5

Trends: Streaming Rate for Premium Content � Average streaming rate has increased 20-40% every year 1600 1400 1200 1000 Kbps 800 600 400 200 0 2006 2007 2008 2009 2010 6

Trends: Per-hour Streaming Cost � Per-hour streaming cost has decreased 15-35% every year 10 9 8 cents/hour 7 6 5 4 3 2 1 0 2006 2007 2008 2009 2010 7

HTTP Chunking � Trend accelerated by switching from proprietary streaming technologies (e.g., Adobe’s FMS) to HTTP Chunking:  Move Networks (2005)  Apple (2008)  Microsoft (2008/2009)  Adobe (2010, 2 nd half) 8

How Does HTTP Chunking Work? … origin CDN2 CDN1 ISP A ISP B http cache Viewer Viewer Viewer 9

HTTP Chunking Advantages � Chunks: immutable, relative large objects (hundreds of KB)  Great for caching � Leverage existing HTTP infrastructure  CDNs  ISP deployed caches  Enterprise http proxies � Low cost and high scale 10

What Does this Mean? � Ad supported premium content  CPM (cost per thousand of ad impressions) for premium content has reached: $20-$40  One ad covers one hour of streaming! � Paid content  $0.99 episode, distribution cost < 3% � Subscription based premium content  Distribution, usually a few percents of total cost  It costs $1.6 per month to stream content to an user watching 2 hours per day � Production & rights costs dominate 11

Quality Matters 12

Quality Matters � Better quality  Increase viewing time  more ad opportunities  Increase retention rate  Protect brand � Quality  Join time  Buffering ratio  Rendering quality  Streaming rate 13

Analysis � Load:  Four channels of a premier video-on demand (VoD) content producer  Four days  Number of sessions (views): 1,176,049  A large live event: ~250,000 concurrent viewers � Metrics  Content length distribution  Viewer Hour Loss (VHL): number of viewer hours lost due to quality issues

Number of Objects 100 150 200 250 300 50 0 VoD Object Length Distribution 10 40 70 100 130 160 190 220 250 [2min, 3min] Short clips: 280 310 340 370 400 430 Object Length 460 490 520 550 580 610 640 Medium clips: [9min, 11min] 680 750 880 1240 1330 2540 2570 [42min, 45min] Full Episodes: 2600 2640 4040 5410 5550 15

Quality Metrics � Buffering Quality (BQ): PlayingTime/(PlayingTime + BufferingTime)  Rendering Quality (RQ): RenderingRate/EncoderRate  Good session  BQ > 95%  RQ > 60%

Analysis Underestimates Quality Impact � For most analysis use BQ only  RQ only a small part of quality issues due to low bit rate (500-700Kbps) � Ignore connection failures

Short Clip (2-3min) Analysis 2 1.8 1.6 Average Playing Time (Minutes) 1.4 1.2 1 0.8 0.6 0.4 0.2 0 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 18

Short Clip (2-3min) Analysis 300000 250000 Number of Sessions 200000 150000 100000 50000 0 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 19

Viewer Hour Gain � D : Average duration of sessions with high quality (0.98 <= quality < 1) � D q : Average duration of sessions with quality = q � N q : Number of sessions with quality = q � Viewer hour gain for sessions with quality q N q x (D – D q ) � Total viewer hour gain ∑ q N q x (D – D q ) Viewer hour loss for 1-2 minute clips: 20

Medium Clip (9-11min) Analysis 8 7 Average Playing Time (Minutes) 6 5 4 3 2 1 0 Viewer hour loss for 9-11min clips: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 21

Full Episodes (42-45min) Analysis 25 Average Playing Time (Minutes) 20 15 10 5 0 • Viewer hour loss for episodes: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 • Viewer hour loss for all content: Quality 22

Large Scale Live Event 35 30 Average Playing Time (Minutes) 25 20 15 10 5 0 Viewer hour loss: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 23

Large Scale Live Event: Engagement Funnel 1 0.9 0.8 0.7 Fraction of Sesions 0<=Q<=0.5 0.6 0.5<=Q<=0.75 0.75<=Q<=0.9 0.5 1 0.99 0.9 <= Q <= 0.95 0.8624 0.4 0.95<=Q<=0.98 0.7364 0.6602 0.98<=Q<=1.0 0.3 0.5509 Quality = 1.0 0.4357 0.2 Continuing 0.3639 0.1 Half people leave due to quality 0 issues All 0-0 0-1 1-3 3-5 5-10 10-20 20-30 Session Duration (Mins) 24

Another Case Study: Live Event Quality Engagement Peak Concurrent Views Total views 151,980 25 minutes 20000 Failed views 13,815 (9%) 0 minutes 15000 Quality impacted 21,584 (14%) 16 minutes 10000 views 5000 Good views 116,581 (77%) 27 minutes 0 Unique viewers 75,328 48 minutes 7:00 PM 7:15 PM 7:30 PM 7:45 PM 8:00 PM 8:15 PM 8:30 PM 8:45 PM 9:00 PM 9:15 PM 9:30 PM 9:45 PM 10:00 PM 10:15 PM 10:30 PM 10:45 PM 11:00 PM 11:15 PM 11:30 PM 11:45 PM 12:00 AM 12:15 AM 12:30 AM 12:45 AM 1:00 AM Failed viewers 1,386 (2%) 0 minutes Quality impacted 14,309 (19%) 30 minutes Total sessions 151,980 viewers Good viewers 59,633 (79%) 51 minutes Unique viewers 73,942 Total viewer hours 58,436 hours Sessions per viewer 1.9 Lost viewer hours 5,134 hours Total viewer hours 58,436 (9%) Viewer with poor quality watch 41% less minutes! 25

Does High Bit Rate Video Help? � Comparing Engagement of low and high bitrates  Viewers watch longer on average on 1500Kbps Average Play Time (minutes) 35 30 10-15% increase 25 minutes 20 15 500 kbps 10 1500 kbps 5 0

Summary � Quality impact:  BQ can impact viewer engagement by up to 40%  Higher bit-rates can increase viewer engagement by up to 15% � Engagement loss due to quality issues: between 4 and 30%  Even a 4% improvement, may offset distribution costs  Ignore other quality issues, like connectivity and media failures 27

Root Cause Analysis 28

Viewers vs. Buffering Quality 250000 100.00% 90.00% 200000 80.00% 70.00% 150000 60.00% Light Period Heavy Period 50.00% 100000 40.00% 30.00% 50000 20.00% 10.00% 0 0.00% 0:00:00 0:15:00 0:30:00 0:45:00 1:00:00 1:15:00 1:30:00 1:45:00 2:00:00 2:15:00 2:30:00 2:45:00 3:00:00 3:15:00 3:30:00 3:45:00 4:00:00 4:15:00 4:30:00 4:45:00 5:00:00 5:15:00 5:30:00 5:45:00 6:00:00 6:15:00 What causes quality issues? Viewers Average Buffering Quality % 29

Root Cause Analysis Last Mile Peering CDN + ISP ISP Viewer � Root cause a quality issue to:  Viewer machine (CPU)  Last mile + ISP (Autonomous System Number)  CDN � Note:  Cannot differentiate between edge and core ISPs  Use only passive measurements, no IP traceroute

Metrics and Definitions � Quality metrics  Buffering quality (BQ)  playing time/(playing time + buffering time)  Rendering quality (RQ)  rendering frame rate/encoded frame rate � Session classification:  Good: (BQ >= 95%) AND (RQ >= 60%)  Low BQ: (BQ < 95%)  Low RQ: (BQ >= 95%) AND (RQ < 60%)

Methodology: Root Causing Viewer Machine � CPU likely to be the issue when:  Rendering quality low  Buffering quality high � Conclude CPU is the issue when session’s  RQ < 60%  BQ > 95%

Quality Issues: Light Period 4% (CPU issue) Network/CDN issues 21.5% Good Low BQ Low RQ 74%

Quality Issues: High Period 6.2% (CPU issue) Network/CDN issues Good 31.5% Low BQ 62% Low RQ

Explaining Buffering Issues � Assume buffering quality issues are either due to:  CDN, or  ISP � Recall: a session has buffering quality issues if  BQ < 95%

Methodology: Root Causing CDN (1/2) � Viewers connected to same ASN but using two CDNs � Intuition: if quality experienced by CDN 1 viewers is significantly lower than of CDN 2 viewers for same ASN, CDN 1 has quality issues CDN 1 Peering Last Mile ISP + ISP Peering ISP CDN 2

Methodology: Root Causing CDN (2/2) � Select all ASNs who have more than 50 sessions for each CDN • If difference between quality of viewers in CDN1 and CDN2 for same ASN is > 10%  Lower quality CDN is root cause at current time