Introduction to Randomized Algorithms: QuickSort Lecture 2 August - PowerPoint PPT Presentation

CS 498ABD: Algorithms for Big Data Introduction to Randomized Algorithms: QuickSort Lecture 2 August 27, 2020 Chandra (UIUC) CS498ABD 1 Fall 2020 1 / 33

Outline Today Randomized Algorithms – Two types Las Vegas Monte Carlo Randomized Quick Sort Chandra (UIUC) CS498ABD 2 Fall 2020 2 / 33

Part I Introduction to Randomized Algorithms Chandra (UIUC) CS498ABD 3 Fall 2020 3 / 33

Randomized Algorithms Input x Output y Deterministic Algorithm Chandra (UIUC) CS498ABD 4 Fall 2020 4 / 33

Randomized Algorithms Input x Output y Deterministic Algorithm random bits r Input x Output y r Randomized Algorithm Chandra (UIUC) CS498ABD 4 Fall 2020 4 / 33

Example: Randomized QuickSort QuickSort ? Pick a pivot element from array 1 Split array into 3 subarrays: those smaller than pivot, those 2 larger than pivot, and the pivot itself. Recursively sort the subarrays, and concatenate them. 3 Randomized QuickSort Pick a pivot element uniformly at random from the array 1 Split array into 3 subarrays: those smaller than pivot, those 2 larger than pivot, and the pivot itself. Recursively sort the subarrays, and concatenate them. 3 Chandra (UIUC) CS498ABD 5 Fall 2020 5 / 33

Example: Randomized Quicksort Recall: QuickSort can take Ω ( n 2 ) time to sort array of size n . = U , 5 , 6 , 3 , 100 R , 2 , - . . , - I , too 2=3 , 0 - - I \ = Of - too 2 3 , . - + Tcu -17 n - Tnk n Chandra (UIUC) CS498ABD 6 Fall 2020 6 / 33

Example: Randomized Quicksort Recall: QuickSort can take Ω ( n 2 ) time to sort array of size n . Theorem Randomized QuickSort sorts a given array of length n in O = O ( n log n ) expected time. witu¥dihG . Chandra (UIUC) CS498ABD 6 Fall 2020 6 / 33

Example: Randomized Quicksort Recall: QuickSort can take Ω ( n 2 ) time to sort array of size n . Theorem Randomized QuickSort sorts a given array of length n in O ( n log n ) expected time. Note: On every input randomized QuickSort takes O ( n log n ) time in expectation. On every input it may take Ω ( n 2 ) time with some small probability. Chandra (UIUC) CS498ABD 6 Fall 2020 6 / 33

Example: Verifying Matrix Multiplication Problem Given three n ⇥ n matrices A , B , C is AB = C ? Chandra (UIUC) CS498ABD 7 Fall 2020 7 / 33

Example: Verifying Matrix Multiplication Problem Given three n ⇥ n matrices A , B , C is AB = C ? Deterministic algorithm: Multiply A and B and check if equal to C . 1 Running time? O ( n 3 ) by straight forward approach. O ( n 2 . 37 ) 2 with fast matrix multiplication (complicated and impractical). " E' ' ' " I an :c . a - Chandra (UIUC) CS498ABD 7 Fall 2020 7 / 33

Example: Verifying Matrix Multiplication Problem Given three n ⇥ n matrices A , B , C is AB = C ? Randomized algorithm: Pick a random n ⇥ 1 vector r . 1 -ux . I Return the answer of the equality ABr = Cr . 2 Running time? 3 It Chandra (UIUC) CS498ABD 8 Fall 2020 8 / 33

Example: Verifying Matrix Multiplication Problem Given three n ⇥ n matrices A , B , C is AB = C ? Randomized algorithm: Pick a random n ⇥ 1 vector r . 1 Return the answer of the equality ABr = Cr . 2 Running time? O ( n 2 ) ! 3 Chandra (UIUC) CS498ABD 8 Fall 2020 8 / 33

Example: Verifying Matrix Multiplication Problem Given three n ⇥ n matrices A , B , C is AB = C ? Randomized algorithm: Pick a random n ⇥ 1 vector r . 1 Return the answer of the equality ABr = Cr . 2 Running time? O ( n 2 ) ! 3 Theorem If AB = C then the algorithm will always say YES. If AB 6 = C then the algorithm will say YES with probability at most 1 / 2 . Can repeat the algorithm 100 times independently to reduce the probability of a false positive to 1 / 2 100 . Chandra (UIUC) CS498ABD 8 Fall 2020 8 / 33

Why randomized algorithms? Many many applications in algorithms, data structures and 1 computer science! In some cases only known algorithms are randomized or 2 randomness is provably necessary. Often randomized algorithms are (much) simpler and/or more 3 e ffi cient. Several deep connections to mathematics, physics etc. 4 . . . 5 Lots of fun! 6 Chandra (UIUC) CS498ABD 9 Fall 2020 9 / 33

Average case analysis vs Randomized algorithms Average case analysis: Fix a deterministic algorithm. 1 Assume inputs comes from a probability distribution. 2 Analyze the algorithm’s average performance over the 3 distribution over inputs. Randomized algorithms: Algorithm uses random bits in addition to input. 1 Analyze algorithms average performance over the given input 2 where the average is over the random bits that the algorithm uses. On each input behaviour of algorithm is random. Analyze 3 worst-case over all inputs of the (average) performance. Chandra (UIUC) CS498ABD 10 Fall 2020 10 / 33

Types of Randomized Algorithms Typically one encounters the following types: Las Vegas randomized algorithms: for a given input x 1 output of algorithm is always correct but the running time is a random variable . In this case we are interested in analyzing the expected running time. Chandra (UIUC) CS498ABD 11 Fall 2020 11 / 33

Types of Randomized Algorithms Typically one encounters the following types: Las Vegas randomized algorithms: for a given input x 1 output of algorithm is always correct but the running time is a random variable . In this case we are interested in analyzing the expected running time. Monte Carlo randomized algorithms: for a given input x the 2 running time is deterministic but the output is random ; correct with some probability. In this case we are interested in analyzing the probability of the correct output (and also the running time). Algorithms whose running time and output may both be random. 3 Chandra (UIUC) CS498ABD 11 Fall 2020 11 / 33

Analyzing Las Vegas Algorithms Deterministic algorithm Q for a problem Π : Let Q ( x ) be the time for Q to run on input x of length | x | . 1 Worst-case analysis: run time on worst input for a given size n . 2 T wc ( n ) = max x : | x | = n Q ( x ) . = it Olntosn ) Chandra (UIUC) CS498ABD 12 Fall 2020 12 / 33

Analyzing Las Vegas Algorithms Deterministic algorithm Q for a problem Π : Let Q ( x ) be the time for Q to run on input x of length | x | . 1 Worst-case analysis: run time on worst input for a given size n . 2 T wc ( n ) = max x : | x | = n Q ( x ) . Randomized algorithm R for a problem Π : Let R ( x ) be the time for Q to run on input x of length | x | . 1 = R ( x ) is a random variable: depends on random bits used by R . 2 E[ R ( x )] is the expected running time for R on x 3 Worst-case analysis: expected time on worst input of size n 4 T rand − wc ( n ) = max x : | x | = n E[ R ( x )] . ⇒ I = Chandra (UIUC) CS498ABD 12 Fall 2020 12 / 33

Analyzing Monte Carlo Algorithms Randomized algorithm M for a problem Π : Let M ( x ) be the time for M to run on input x of length | x | . 1 For Monte Carlo, assumption is that run time is deterministic. Let Pr[ x ] be the probability that M is correct on x . 2 Pr[ x ] is a random variable: depends on random bits used by M . 3 Worst-case analysis: success probability on worst input 4 P rand − wc ( n ) = min x : | x | = n Pr[ x ] . Chandra (UIUC) CS498ABD 13 Fall 2020 13 / 33

Part II Randomized Quick Sort Chandra (UIUC) CS498ABD 14 Fall 2020 14 / 33

Randomized QuickSort Randomized QuickSort Pick a pivot element uniformly at random from the array. 1 Split array into 3 subarrays: those smaller than pivot, those 2 larger than pivot, and the pivot itself. Recursively sort the subarrays, and concatenate them. 3 I 1,3 , 5 , 12,14 , 16 , 18,19 array: 16, 12, 14, 20, 5, 3, 18, 19, 1 1 as its position , 20 of an element rank in the sorted aerag . Chandra (UIUC) CS498ABD 15 Fall 2020 15 / 33

Analysis What events to count? Number of Comparisions. Chandra (UIUC) CS498ABD 16 Fall 2020 16 / 33

Analysis What events to count? Number of Comparisions. What is the probability space? All the coin tosses at all levels and parts of recursion. Chandra (UIUC) CS498ABD 16 Fall 2020 16 / 33

Analysis What events to count? Number of Comparisions. What is the probability space? All the coin tosses at all levels and parts of recursion. Too Big!! Chandra (UIUC) CS498ABD 16 Fall 2020 16 / 33

Analysis What events to count? Number of Comparisions. What is the probability space? All the coin tosses at all levels and parts of recursion. Too Big!! What random variables to define? What are the events of the algorithm? Chandra (UIUC) CS498ABD 16 Fall 2020 16 / 33

Analysis via Recurrence Given array A of size n , let Q ( A ) be number of comparisons of 1 randomized QuickSort on A . Note that Q ( A ) is a random variable. 2 Let A i left and A i right be the left and right arrays obtained if rank i 3 element chosen as pivot. Let X i be indicator random variable, which is set to 1 if pivot is t.iq#I of rank i in A , else zero. n ⇣ ⌘ X Q ( A i left ) + Q ( A i Q ( A ) = n + X i · right ) . T i =1 E random variables Chandra (UIUC) CS498ABD 17 Fall 2020 17 / 33