Asymptotic Algorithm Analysis We can analyse an algorithm without - PowerPoint PPT Presentation

Topic 6: Algorithm Analysis & Sorting 1 (Version of November 14, 2011) Pierre Flener Computing Science Division Department of Information Technology Uppsala University Sweden Course 1DL201: Program Construction and Data Structures 1 Based on original slides by John Hamer and Yves Deville, with some figures from the CLRS textbook (which are c � The MIT Press, 2009)

Asymptotic Algorithm Analysis We can analyse an algorithm without needing to run it, and thus gain some understanding of its likely performance. Road Map Asymptotic Algorithm This analysis can be done at design time, before the Analysis program is written. Even if the analysis is approximate, Insertion Sort performance problems may be detected. Merge Sort Master The notation used in the analysis is helpful in documenting Method software libraries. It allows programs using such libraries to Quick Sort be analysed without requiring analysis of the library source Accumulator Introduction code (which is often not available). Stable Sorting We will mostly analyse the runtime performance. Road Map Revisited The same principles apply to memory consumption. We speak of time complexity and space complexity. Course 1DL201 - 6 - Program Construction and Data Structures

The Θ Notation The Θ notation is used to denote a set of functions that Road Map increase at the same rate (within some constant bound). Asymptotic Algorithm Analysis Formally, Θ ( g ( n )) is the set of all functions f ( n ) that are Insertion bounded below by c 1 · g ( n ) � 0 and above by c 2 · g ( n ) , for Sort some constants c 1 > 0 and c 2 > 0, when n gets sufficiently Merge Sort large, that is, when n is at least some constant n 0 > 0. Master Method Quick Sort The function g ( n ) in Θ ( g ( n )) is called a complexity function. Accumulator Introduction Stable We write f ( n ) = Θ ( g ( n )) when we mean f ( n ) 2 Θ ( g ( n )) . Sorting Road Map Revisited The Θ notation is used to give asymptotically tight bounds. Course 1DL201 - 12 - Program Construction and Data Structures

Terminology Let lg x denote log 2 x , let k � 2 be a constant, and let variable n denote the input size: Road Map Function Growth Rate Asymptotic Algorithm 1 constant Analysis lg n logarithmic sub-linear Insertion lg 2 n Sort log-squared Merge Sort n linear Master n · lg n Method polynomial n 2 quadratic Quick Sort n 3 Accumulator cubic Introduction k n exponential exponential Stable Sorting n ! super-exponential Road Map n n Revisited + From now on (and in the homeworks), we use Θ ( 1 ) instead of introducing constants such as t 0 and t add . Course 1DL201 - 13 - Program Construction and Data Structures

Example Theorem: n 2 + 5 · n + 10 = Θ ( n 2 ) . Proof: We need to choose constants c 1 > 0, c 2 > 0, Road Map and n 0 > 0 such that Asymptotic Algorithm 0  c 1 · n 2  n 2 + 5 · n + 10  c 2 · n 2 Analysis Insertion Sort for all n � n 0 . Dividing by n 2 (assuming n > 0) gives Merge Sort Master Method 0  c 1  1 + 5 n + 10 n 2  c 2 Quick Sort Accumulator Introduction The “sandwiched” term, 1 + 5 n + 10 n 2 , gets smaller as n grows. Stable It peaks at 16 for n = 1, so we can pick n 0 = 1 and c 2 = 16. Sorting Road Map It drops to 6 for n = 2 and becomes close to 1 for n = 1000. Revisited It never gets less than 1, so we can pick c 1 = 1. Exercise: Prove that 5 · n 3 + 7 · n 2 � 3 · n + 4 6 = Θ ( n 2 ) . Course 1DL201 - 14 - Program Construction and Data Structures

Keep Complexity Functions Simple While it is formally (and trivially) correct to say that n 2 + 5 · n + 10 = Θ ( n 2 + 5 · n + 10 ) , the whole purpose of Road Map Asymptotic the Θ notation is to work with simple expressions. Algorithm Analysis Thus, we often do not expect any arbitrary factors or Insertion lower-order terms inside a complexity function. Sort Merge Sort We can simplify complexity functions by: Master Method Setting all constant factors to 1. Quick Sort Dropping all lower-order terms. Accumulator Introduction Stable 1 1 Since log b x = Sorting log c b · log c x , where log c b is a constant factor Road Map (when the bases b and c are constants), we shall use lg x Revisited in complexity functions. Course 1DL201 - 15 - Program Construction and Data Structures

Variations on Θ : The O and Ω Notations Variants of Θ include O (“big-Oh”), which drops the lower bound, and Ω (“big-Omega”), which drops the upper bound: Road Map Asymptotic Algorithm Analysis Insertion Sort Merge Sort Master Method Quick Sort Examples: Any linear function a · n + b is in O ( n 2 ) , O ( n 3 ) , Accumulator and so on, but not in Θ ( n 2 ) , Θ ( n 3 ) , and so on. Any Introduction quadratic function a · n 2 + b · n + c is in Ω ( n ) . We use O to Stable Sorting give an upper bound on a function, and Ω to give a lower Road Map Revisited bound, but no claims are made about how tight these bounds are. We use Θ to give a tight bound, namely when the upper and lower bounds are the same. Course 1DL201 - 17 - Program Construction and Data Structures

Application of a Pre-Established Formula Theorem 1 (proof omitted): If, for some constants a and b : Road Map Asymptotic ( Θ ( 1 ) if n  b Algorithm C ( n ) = Analysis a · C ( n � 1 ) + Θ ( 1 ) if n > b Insertion Sort Merge Sort then the closed form of the recurrence is: Master Method ( Θ ( n ) if a = 1 C ( n ) = Quick Sort Θ ( a n ) if a > 1 Accumulator Introduction Stable Sorting Another pre-established formula is in the Master Theorem: Road Map Revisited page 44 Course 1DL201 - 24 - Program Construction and Data Structures

The Master Method and Master Theorem From now on, we will ignore the base cases of a recurrence. Road Map The closed form for a recurrence T ( n ) = a · T ( n / b ) + f ( n ) Asymptotic reflects the “battle” between the two terms in the sum. Algorithm Analysis Think of a · T ( n / b ) as the process of “distributing the work Insertion Sort out” to f ( n ) , where the actual work is done. Merge Sort Theorem 2 (known as the Master Theorem, proof omitted): Master Method 1 If f ( n ) is dominated by n log b a (see the next page), Quick Sort then T ( n ) = Θ ( n log b a ) . Accumulator Introduction 2 If f ( n ) and n log b a are balanced (if f ( n ) = Θ ( n log b a ) ), Stable then T ( n ) = Θ ( n log b a · lg n ) . Sorting Road Map 3 If f ( n ) dominates n log b a and if the regularity condition Revisited (see the next page) holds, then T ( n ) = Θ ( f ( n )) . Course 1DL201 - 44 - Program Construction and Data Structures

Dominance and the Regularity Condition The three cases of the Master Theorem depend on comparing f ( n ) to n log b a . However, it is not sufficient for f ( n ) Road Map to be “just a bit” smaller or bigger than n log b a . Cases 1 Asymptotic Algorithm and 3 only apply when there is a polynomial difference Analysis between these functions, that is when the ratio between the Insertion Sort dominator and the dominee is asymptotically larger than the Merge Sort polynomial n ✏ for some constant ✏ > 0. Master Method Example: n 2 is polynomially larger than both n 1 . 5 and lg n . Quick Sort Accumulator Introduction Counter-Example: n · lg n is not polynomially larger than n . Stable Sorting In Case 3, a regularity condition requires a · f ( n / b )  c · f ( n ) Road Map Revisited for some constant c < 1 and all sufficiently large n . (All the f functions in this course will satisfy this condition.) Course 1DL201 - 45 - Program Construction and Data Structures

Gaps in the Master Theorem The Master Theorem does not cover all possible Road Map recurrences of the form T ( n ) = a · T ( n / b ) + f ( n ) : Asymptotic Cases 1 and 3: The difference between f ( n ) and n log b a Algorithm Analysis might not be polynomial. Insertion Counter-Example: The Master Theorem does not Sort apply to the recurrence T ( n ) = 2 · T ( n / 2 ) + n · lg n , Merge Sort despite it having the proper form. We have a = 2 = b , Master Method so we need to compare f ( n ) = n · lg n to Quick Sort n log b a = n 1 = n . Clearly, f ( n ) = n · lg n > n for large Accumulator Introduction enough n , but the ratio f ( n ) / n is lg n , which is asymptotically less than the polynomial n ✏ for any Stable Sorting constant ✏ > 0, so we are not in Case 3. Road Map Revisited Case 3: The regularity condition might not hold. Course 1DL201 - 46 - Program Construction and Data Structures

Common Cases of the Master Theorem n log b a a b f ( n ) Case T ( n ) Road Map Θ ( 1 ) 2 Θ ( lg n ) Asymptotic Algorithm Θ ( lg n ) none Θ (?) n 0 1 2 Analysis Θ ( n · lg n ) 3 Θ ( n · lg n ) Insertion Θ ( n k ) , with k > 0 Θ ( n k ) Sort 3 Merge Sort Θ ( 1 ) 1 Θ ( n ) Master Θ ( lg n ) Θ ( n ) 1 Method n 1 2 2 Θ ( n ) 2 Θ ( n · lg n ) Quick Sort Accumulator Θ ( n · lg n ) none Θ (?) Introduction Θ ( n k ) , with k > 1 Θ ( n k ) 3 Stable Sorting Road Map Revisited (This table can only be used for looking up a closed form, but it cannot be referred to in the homeworks or exams.) Course 1DL201 - 47 - Program Construction and Data Structures