CS 573: Algorithms, Fall 2013 Randomized Algorithms II – High Probability Part I Lecture 10 Movie... September 26, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 51 Sariel (UIUC) CS573 2 Fall 2013 2 / 51 Binomial distribution X n = numbers of heads when flipping a coin n times. Claim Part II = ( n i ) � � X n = i 2 n . Pr Understanding the binomial � n � n ! Where: = ( n − k )! k ! . k � n � Indeed, is the number of ways to choose i elements out of n distribution i elements (i.e., pick which i coin flip come up heads). Each specific such possibility (say 0100010 ... ) had probability 1 / 2 n . Sariel (UIUC) CS573 3 Fall 2013 3 / 51 Sariel (UIUC) CS573 4 Fall 2013 4 / 51
Massive randomness.. Is not that random. Massive randomness.. Is not that random. Consider flipping a fair coin n times independently, head given 1 , tail gives zero. How many heads? ...we get a binomial distribution. This is known as concentration of mass . This is a very special case of the law of large numbers . Sariel (UIUC) CS573 5 Fall 2013 5 / 51 Sariel (UIUC) CS573 6 Fall 2013 6 / 51 Side note... Massive randomness.. Is not that random. Law of large numbers (weakest form)... Intuitive conclusion Randomized algorithm are unpredictable in the tactical level, but very predictable in the strategic level. Informal statement of law of large numbers For n large enough, the middle portion of the binomial distribution looks like (converges to) the normal/Gaussian distribution. Sariel (UIUC) CS573 7 Fall 2013 7 / 51 Sariel (UIUC) CS573 8 Fall 2013 8 / 51
Show that QuickSort running time is O ( n log n ) QuickSort picks a pivot, splits into two subproblems, and 1 continues recursively. Track single element in input. 2 Part III Game ends, when this element is alone in subproblem. 3 Show every element in input, participates ≤ 32 ln n rounds 4 QuickSort with high probability (with high enough probability). E i : event i th element participates > 32 ln n rounds. 5 C QS : number of comparisons performed by QuickSort . 6 Running time O ( C QS ) . 7 Probability of failure is 8 � � � � i E i ] ≤ � n α = Pr C QS ≥ 32 n ln n ≤ Pr [ � . i = 1 Pr E i ... by the union bound. Sariel (UIUC) CS573 9 Fall 2013 9 / 51 Sariel (UIUC) CS573 10 Fall 2013 10 / 51 Show that QuickSort running time is O ( n log n ) Proving that an element... ... participates in small number of rounds. Probability of failure is 1 � � � � � n n : number of elements in input for QuickSort . α = Pr C QS ≥ 32 n ln n ≤ Pr [ i E i ] ≤ . � i = 1 Pr E i 1 x : Arbitrary element x in input. Union bound : for any two events A and B : 2 2 Pr [ A ∪ B ] ≤ Pr [ A ] + Pr [ B ] . S 1 : Input. 3 Assume: Pr [ E i ] ≤ 1 / n 3 . S i : input to i th level recursive call that include x . 3 4 � � 1 1 x lucky in j th iteration, if balanced split... � n ≤ � n Bad probability... α ≤ n 3 = n 2 . i = 1 Pr E i 5 4 i = 1 | S j + 1 | ≤ ( 3 / 4 ) | S j | and | S j \ S j + 1 | ≤ ( 3 / 4 ) | S j | = ⇒ QuickSort performs ≤ 32 n ln n comparisons, w.h.p. 5 Y j = 1 ⇐ ⇒ x lucky in j th iteration. 6 = ⇒ QuickSort runs in O ( n log n ) time, with high probability. 6 � � = 1 Pr Y j 2 . 7 Observation: Y 1 , Y 2 , . . . , Y m are independent variables. 8 x can participate ≤ ρ = log 4 / 3 n ≤ 3 . 5 ln n rounds. 9 10 ...since | S j | ≤ n ( 3 / 4 ) # of lucky iteration in 1 ... j . 11 If ρ lucky rounds in first k rounds = ⇒ | S k | ≤ ( 3 / 4 ) ρ n ≤ 1 . Sariel (UIUC) CS573 11 Fall 2013 11 / 51 Sariel (UIUC) CS573 12 Fall 2013 12 / 51
Proving that an element... Proving that an element... ... participates in small number of rounds. ... participates in small number of rounds. Assume the following: 1 Lemma In M coin flips: Pr [# heads ≤ M / 4 ] ≤ exp ( − M / 8 ) . Brain reset! 1 Set M = 32 ln n ≥ 8 ρ . 2 Q: How many rounds x participates in = how many coin flips till 2 Pr [ Y j = 0 ] = Pr [ Y j = 1 ] = 1 / 2 . 3 one gets ρ heads? Y 1 , Y 2 , . . . , Y M are independent. 4 A: In expectation, 2 ρ times. 3 = ⇒ probability ≤ ρ ≤ M / 4 ones in Y 1 , . . . , Y M is 5 � − M � ≤ exp ( − ρ ) ≤ 1 ≤ exp n 3 . 8 = ⇒ probability x participates in M recursive calls of 6 QuickSort ≤ 1 / n 3 . Sariel (UIUC) CS573 13 Fall 2013 13 / 51 Sariel (UIUC) CS573 14 Fall 2013 14 / 51 Proving that an element... Alternative proof of high probability of QuickSort ... participates in small number of rounds. T : n items to be sorted. 1 t ∈ T : element. 2 X i : the size of subproblem in i th level of recursion containing t . 3 n input elements. Probability depth of recursion in QuickSort 1 � � � ≤ 1 3 4 X i − 1 + 1 2 X i − 1 ≤ 7 X 0 = n , and E X i � X i − 1 8 X i − 1 . > 32 ln n is ≤ ( 1 / n 3 ) ∗ n = 1 / n 2 . 4 � 2 � � � � � � � Result: ∀ random variables E X = E y X � Y = y . 2 E 5 � Theorem � � � � � � � � � 7 = E y � X i − 1 = y ≤ E X i − 1 = y = X i X i 8 y E E 6 � With high probability (i.e., 1 − 1 / n 2 ) the depth of the recursion of � i n . � i QuickSort is ≤ 32 ln n . Thus, with high probability, the running � � � � 7 7 7 ≤ E [ X 0 ] = X i − 1 8 E 8 8 time of QuickSort is O ( n log n ) . Same result holds for MatchNutsAndBolts . 3 Sariel (UIUC) CS573 15 Fall 2013 15 / 51 Sariel (UIUC) CS573 16 Fall 2013 16 / 51
Alternative proof of high probability of QuickSort � M n ≤ � � � 7 1 1 M = 8 log 8 / 7 n : µ = E ≤ n 8 n = n 7 . X M 1 8 Markov’s Inequality: For a non-negative variable X , and t > 0 , 2 we have: Part IV � � E [ X ] Pr X ≥ t ≤ . t Chernoff inequality By Markov’s inequality: 3 � � ≤ E [ X M ] ≤ 1 t participates � � ≤ Pr X M ≥ 1 n 7 . Pr > M recursive calls 1 Probability any element of input participates > M recursive 4 calls ≤ n ( 1 / n 7 ) ≤ 1 / n 6 . Sariel (UIUC) CS573 17 Fall 2013 17 / 51 Sariel (UIUC) CS573 18 Fall 2013 18 / 51 Preliminaries Chernoff inequality X , Y : Random variables are independent if ∀ x , y : 1 Theorem (Chernoff inequality) � � � � � � ( X = x ) ∩ ( Y = y ) = Pr X = x · Pr Y = y . X 1 , . . . , X n : n independent random variables, such that Pr Pr [ X i = 1 ] = Pr [ X i = − 1 ] = 1 2 , for i = 1 , . . . , n . Let Y = � n The following is easy to prove: i = 1 X i . Then, for any ∆ > 0 , we have 2 Claim � � � � − ∆ 2 / 2 n Y ≥ ∆ ≤ exp . Pr If X and Y are independent = ⇒ E [ XY ] = E [ X ] E [ Y ] . ⇒ Z = e X and W = e Y are independent. = Sariel (UIUC) CS573 19 Fall 2013 19 / 51 Sariel (UIUC) CS573 20 Fall 2013 20 / 51
Proof of Chernoff inequality Proof of Chernoff inequality Continued... Fix arbitrary t > 0 : 2 e − t = e t + e − t = 1 2 e t + 1 � � � � � � Pr Y ≥ ∆ = Pr tY ≥ t ∆ = Pr exp ( tY ) ≥ exp ( t ∆) � � exp ( tX i ) E 2 � � exp ( tY ) ≤ E 1 ! + t 2 2 ! + t 3 � � 1 1 + t , = 3 ! + · · · exp ( t ∆) 2 1 ! + t 2 2 ! − t 3 + 1 � 1 − t � 3 ! + · · · 2 = 1 + t 2 t 2 k 2 ! + + · · · + ( 2 k )! + · · · . However: ( 2 k )! = k !( k + 1 )( k + 2 ) · · · 2 k ≥ k ! 2 k . � i ∞ t 2 i ∞ t 2 i ∞ � t 2 � t 2 1 � � � � � � exp ( tX i ) = ( 2 i )! ≤ 2 i ( i !) = ≤ = ≤ exp . E i ! 2 2 i = 0 i = 0 i = 0 Sariel (UIUC) CS573 21 Fall 2013 21 / 51 Sariel (UIUC) CS573 22 Fall 2013 22 / 51 � �� �� �� � n � � � � � exp ( tY ) = E = E exp ( tX i ) = exp ( tX i ) ≤ exp tX i E E i = 1 i i Chernoff inequality... Chernoff inequality... � � � nt 2 � exp ( tY ) exp � nt 2 ≤ E � � � 2 Pr Y ≥ ∆ ≤ exp ( t ∆) = exp − t ∆ . ...what it really says symmetry exp ( t ∆) 2 Set t = ∆ / n : � 2 − ∆ 2 � ∆ − ∆ � � Corollary n � � = exp Y ≥ ∆ ≤ exp n ∆ . Pr By theorem: 2 n 2 n Let X 1 , . . . , X n be n independent random variables, such that Pr [ X i = 1 ] = Pr [ X i = − 1 ] = 1 2 , for i = 1 , . . . , n . Let � n � − ∆ 2 n n � � � � � � Y = � n i � � i = 1 X i . Then, for any ∆ > 0 , we have Y ≥ ∆ = Y = i = 2 n ≤ exp , Pr Pr 2 n i =∆ i = n / 2 +∆ / 2 − ∆ 2 � � � � Pr | Y | ≥ ∆ ≤ 2 exp . 2 n Sariel (UIUC) CS573 23 Fall 2013 23 / 51 Sariel (UIUC) CS573 24 Fall 2013 24 / 51
Recommend
More recommend
