Developing*a*Predic0ve*Model*for** - PowerPoint PPT Presentation

Developing*a*Predic0ve*Model*for** Internet*Video*Quality9of9Experience* Athula*Balachandran ,"Vyas"Sekar," Aditya"Akella,"Srinivasan"Seshan," Ion"Stoica,"Hui"Zhang" 1"

QoE" ! "$$$"" CDN* Be@er"" $$$" Quality" Video" Content* Users* Providers* Higher" Engagement" The*QoE*model* 2" "Diagram"courtesy:"Prof."Ramesh"Sitaraman,"IMC"2012"

Why"do"we"need"a"QoE"model?" AdapLng"video"bitrates" quicker" Picking"the"best"server" Comparing"CDNs" 3"

TradiLonal"Video"Quality"Metrics" Subjec0ve*Scores* (e.g.,*Mean*Opinion*Score)* User"studies"not"representaLve" Objec0ve*Scores* of"“in[the[wild”"experience" *(e.g.,*Peak*Signal**to*Noise*Ra0o)* Does"not"capture"new"effects"" (e.g.,"buffering,"switching" bitrates)" 4"

Internet"Video"is"a"new"ball"game" Subjec0ve*Scores* (e.g.,*Mean*Opinion* Score)* Engagement* (e.g.,*frac0on*of*video*viewed)* Objec0ve*Scores* *(e.g.,*Peak*Signal**to*Noise*Ra0o)* Quality*metrics* 5"

Commonly"used"Quality"Metrics" Join"Time" Buffering"raLo" Rate"of"switching" Rate"of"buffering" Average"Bitrate" 6"

Which"metric"should"we"use?" Today:" Subjec0ve*Scores* (e.g.,*Mean*Opinion* QualitaLve" Score)* Single[metric" Engagement* (e.g.,*frac0on*of*video*viewed)* Objec0ve*Scores* *(e.g.,*Peak*Signal**to*Noise*Ra0o)* Quality*metrics* *Buffering*Ra0o,*Average*bitrate?* 7"

Unified"and"QuanLtaLve"QoE"Model" Subjec0ve*Scores* (e.g.,*Mean*Opinion* Score)* Engagement* (e.g.,*frac0on*of*video*viewed)* Objec0ve*Scores* *(e.g.,*Peak*Signal**to*Noise*Ra0o)* Quality*metrics* *Buffering*Ra0o,*Average*bitrate?* ƒ"( Buffering*Ra0o,*Average*bitrate,…)* 8"

Outline" " • What*makes*this*hard?* • Our"approach" " • Conclusion" " 9"

Complex"Engagement[to[metric"RelaLonships" """"Engagement" Non[monotonic" Engagement" """""""""""""""Average"bitrate" """Engagement" Quality"Metric" Threshold" Ideal"Scenario" """""""""""Rate"of"switching" [Dobrian"et"al."Sigcomm"2011]" 10" "

Complex"Metric"Interdependencies" """""""Join"Time" Join"Time" Bitrate" Rate"of" Rate"of" """""""Average"bitrate" buffering" switching" Rate"of"buffering" Buffering"raLo" """""""Average"bitrate" 11"

Confounding"Factors" Live" Type"of"Video" VOD" Confounding"Factors" Live*and*Video* """"""""CDF""("%"of"users)" on*Demand* can"affect:" (VOD)*sessions** 1) Engagement" have*different* viewing* paUerns.* """"""" "Engagement" 12"

Confounding"Factors" Live" Type"of"Video" VOD" Confounding"Factors" """"""""CDF""("%"of"users)" can"affect:" Live*and*Video*on* Demand*(VOD)* 1) Engagement" sessions** 2) Quality"Metrics" had*different*join* 0me*distribu0on.* """"""" "Join"Time" 13"

Confounding"Factors" Type"of"Video" DSL/Cable" ConnecLvity" Wireless"(3G/4G)" Confounding"Factors" Users*on* can"affect:" wireless* """""""Engagement" 1) Engagement" connec0vity* 2) Quality"Metrics" were* 3) Quality"Metric" ! " more*tolerant* to*rate*of* Engagement" buffering.* """"""""""Rate"of"buffering" 14"

Confounding"Factors" Popularity" Device" Type"of"Video" LocaLon" ConnecLvity" Time"of"day" Day"of"week" Need"systemaLc"approach"to"" idenLfy"and"incorporate"confounding"factors" 15"

Summary"of"Challenges" 1. Capture"complex"engagement[to[metric" relaLonships"and"metric[to[metric" dependencies." 2. IdenLfy"confounding"factors" 3. Incorporate"confounding"factors" 16"

Outline" " • What"makes"this"hard?" • Our*approach* " • Conclusion" " 17"

Challenge"1:"Capture"complex"relaLonships" 18"

Cast"as"a"Learning"Problem" Quality"Metrics" Engagement" MACHINE"LEARNING" QoE"Model" Decision*Trees*performed*the*best.* Accuracy*of*40%*for*predic0ng*within*a*10%*bucket.* 19"

Challenge"2:"IdenLfy"the"confounding"factors" 20"

Test"PotenLal"Factors" ""Confounding" """""""Factors" Quality"Metrics" Engagement" 21"

Test"PotenLal"Factors" Test"1:"RelaLve"InformaLon"Gain" ""Confounding" """""""Factors" Quality"Metrics" Engagement" 22"

Test"PotenLal"Factors" Test"1:"RelaLve"InformaLon"Gain" ""Confounding" Test"2:"Decision"Tree"Structure" """""""Factors" Test"3:"Tolerance"Level" Quality"Metrics" Engagement" 23"

IdenLfying"Key"Confounding"Factors" Factor" RelaLve" Decision"Tree" Tolerance" InformaLon" Structure" Level" Gain" Type"of"video" ✓ " ✓ " ✓ " Popularity" ✗ " ✗ " ✗ " LocaLon" ✗ " ✗ " ✗ " Device"" ✗ " ✓ " ✓ " ConnecLvity" ✗ " ✗ " ✓ " Time"of"day" ✗ " ✗ " ✓ " Day"of"week" ✗ " ✗ " ✗ " VOD users on different devices have different levels of tolerance for rate of buffering and average bitrate 24"

IdenLfying"Key"Confounding"Factors" We are doing feature selection here: Factor" RelaLve" Decision"Tree" Tolerance" InformaLon" Structure" Level" Gain" Type*of*video* ✓ " ✓ " ✓ " Popularity" ✗ " ✗ " ✗ " LocaLon" ✗ " ✗ " ✗ " Device** ✗ " ✓ " ✓ " Connec0vity* ✗ " ✗ " ✓ " Time*of*day* ✗ " ✗ " ✓ " Day"of"week" ✗ " ✗ " ✗ " 25"

Challenge"3:"Incorporate"the"confounding"factors" 26"

Refine"the"Model" Adding*as*a*feature* SpliZng*the*data* Confounding"" Confounding"" Confounding"" """Factors"1" """Factors"2" """Factors"3" Confounding" e.g.,"Live,"Mobile" e.g.,"VOD,"Mobile" e.g.,"VOD,"TV" """""Factors" Quality"" Engagement" Quality" Quality" Quality" Metrics" "Engmnt" "Engmnt" "Engmnt" Metrics" Metrics" Metrics" ML" ML" ML" MACHINE"LEARNING" Model"1" Model"3" Model"2" QoE"Model" QoE"Model" 27"

Comparing"Candidate"SoluLons" Final"Model:"CollecLon"of"decision"trees" Final"Accuracy["70%"( c.f.$ 40%)"for"10%"buckets" 28"

Summary"of"Our"Approach"" 1. Capture"complex"engagement[to[metric" relaLonships"and"metric[to[metric" dependencies" " ! "Use"Machine"Learning" 2. IdenLfy"confounding"factors"" """"" ! "Tests"" 3."Incorporate"confounding"factors"" " ! "Split" 29"