Algorithm Analysis As soon as an Analytic Engine exists, it will - PowerPoint PPT Presentation

A strikingly modern thought Algorithm Analysis “ As soon as an Analytic Engine exists, it will necessarily guide the future Part I course of the science. Whenever any result is sought by its aid, the question will arise—By what course of calculation can these results be arrived at by the machine in the shortest time? ” — Charles Babbage (1864) Tyler Moore CS 2123, The University of Tulsa how many times do you have to turn the crank? Some slides created by or adapted from Dr. Kevin Wayne. For more information see http://www.cs.princeton.edu/~wayne/kleinberg-tardos . Some slides adapted from Dr. Steven Skiena. For more information see http://www.algorist.com Analytic Engine 3 3 / 28 Brute force Polynomial running time Brute force. For many nontrivial problems, there is a natural brute-force Desirable scaling property. When the input size doubles, the algorithm search algorithm that checks every possible solution. should only slow down by some constant factor C . ・ Typically takes 2 n time or worse for inputs of size n . ・ Unacceptable in practice. Def. An algorithm is poly-time if the above scaling property holds. There exists constants c > 0 and d > 0 such that on every input of size n, its running time is bounded choose C = 2 d by c n d primitive computational steps. von Neumann Nash Gödel Cobham Edmonds Rabin (1953) (1955) (1956) (1964) (1965) (1966) 4 5 4 / 28 5 / 28

Polynomial running time Worst-case analysis Worst case. Running time guarantee for any input of size n . We say that an algorithm is efficient if has a polynomial running time. ・ Generally captures efficiency in practice. Justification. It really works in practice! ・ Draconian view, but hard to find effective alternative. ・ In practice, the poly-time algorithms that people develop have low constants and low exponents. ・ Breaking through the exponential barrier of brute force typically Exceptions. Some exponential-time algorithms are used widely in practice exposes some crucial structure of the problem. because the worst-case instances seem to be rare. Exceptions. Some poly-time algorithms do have high constants and/or exponents, and/or are useless in practice. Map graphs in polynomial time Mikkel Thorup Q. Which would you prefer 20 n 100 vs. n 1 + 0.02 ln n ? Department of Computer Science, University of Copenhagen Universitetsparken 1, DK-2100 Copenhagen East, Denmark mthorup@diku.dk Abstract Chen,Grigni, andPapadimitriou(WADS’97andSTOC’98) have introduced a modified notion of planarity, where two faces are considered adjacent if they share at least one point. The corresponding abstract graphs are called map graphs . Chen et.al. raised the question of whether map graphs can be recognized in polynomial time. They showed that the decision problem is in NP and presented a polynomial time algorithm for the special case where we allow at most 4 faces to intersect in any point — if only 3 are allowed to intersect in a point, we get the usual planar graphs. simplex algorithm Linux grep k-means algorithm Chen et.al. conjectured that map graphs can be recognized in polynomial time, and in this paper, their conjectureis settled affirmatively. 6 7 6 / 28 7 / 28 Types of analyses Why it matters Worst case. Running time guarantee for any input of size n . Ex. Heapsort requires at most 2 n log 2 n compares to sort n elements. Probabilistic. Expected running time of a randomized algorithm. Ex. The expected number of compares to quicksort n elements is ~ 2 n ln n . Amortized. Worst-case running time for any sequence of n operations. Ex. Starting from an empty stack, any sequence of n push and pop operations takes O ( n ) operations using a resizing array. Average-case. Expected running time for a random input of size n . Ex. The expected number of character compares performed by 3-way radix quicksort on n uniformly random strings is ~ 2 n ln n . Also. Smoothed analysis, competitive analysis, ... 8 9 8 / 28 9 / 28

Big-Oh notation Notational abuses Upper bounds. T ( n ) is O ( f ( n )) if there exist constants c > 0 and n 0 ≥ 0 Equals sign. O ( f ( n )) is a set of functions, but computer scientists often write such that T ( n ) ≤ c · f ( n ) for all n ≥ n 0 . T ( n ) = O ( f ( n )) instead of T ( n ) ∈ O ( f ( n )) . c · f ( n ) Ex. T ( n ) = 32 n 2 + 17 n + 1 . Ex. Consider f ( n ) = 5 n 3 and g ( n ) = 3 n 2 . T ( n ) ・ T ( n ) is O ( n 2 ) . ・ We have f ( n ) = O ( n 3 ) = g ( n ). choose c = 50, n 0 = 1 ・ T ( n ) is also O ( n 3 ) . ・ Thus, f ( n ) = g ( n ) . ・ T ( n ) is neither O ( n ) nor O ( n log n ) . n 0 n Domain. The domain of f ( n ) is typically the natural numbers { 0, 1, 2, … } . Typical usage. Insertion makes O ( n 2 ) compares to sort n elements. ・ Sometimes we restrict to a subset of the natural numbers. Other times we extend to the reals. Nonnegative functions. When using big-Oh notation, we assume that the T ( n ) Alternate definition. T ( n ) is O ( f ( n )) if functions involved are (asymptotically) nonnegative. lim sup f ( n ) < � . n �� Bottom line. OK to abuse notation; not OK to misuse it. 11 12 11 / 28 12 / 28 Big-Omega notation Big-Theta notation Lower bounds. T ( n ) is Ω ( f ( n )) if there exist constants c > 0 and n 0 ≥ 0 Tight bounds. T ( n ) is Θ ( f ( n )) if there exist constants c 1 > 0 , c 2 > 0 , and n 0 ≥ 0 such that T ( n ) ≥ c · f ( n ) for all n ≥ n 0 . such that c 1 · f ( n ) ≤ T ( n ) ≤ c 2 · f ( n ) for all n ≥ n 0 . c 2 · f ( n ) T ( n ) T ( n ) Ex. T ( n ) = 32 n 2 + 17 n + 1 . Ex. T ( n ) = 32 n 2 + 17 n + 1 . c · f ( n ) c 1 · f ( n ) ・ T ( n ) is both Ω ( n 2 ) and Ω ( n ) . ・ T ( n ) is Θ ( n 2 ) . choose c = 32, n 0 = 1 choose c 1 = 32, c 2 = 50, n 0 = 1 ・ T ( n ) is neither Ω ( n 3 ) nor Ω ( n 3 log n ) . ・ T ( n ) is neither Θ ( n ) nor Θ ( n 3 ) . n 0 n n 0 n Typical usage. Any compare-based sorting algorithm requires Ω ( n log n ) compares in the worst case. Typical usage. Mergesort makes Θ ( n log n ) compares to sort n elements. Meaningless statement. Any compare-based sorting algorithm requires at least O( n log n ) compares in the worst case. 13 14 13 / 28 14 / 28

Big Oh Examples Big Omega Examples Definition Definition T ( n ) is O ( f ( n )) if there exist constants c > 0 and n 0 ≥ 0 such that T ( n ) is Ω( f ( n )) if there exist constants c > 0 and n 0 ≥ 0 such that T ( n ) ≤ c · f ( n ) for all n ≥ n 0 . T ( n ) ≥ c · f ( n ) for all n ≥ n 0 . 1 3 n 2 + 4 n + 6 = O ( n 2 )? 1 3 n 2 + 4 n + 6 = Ω( n 2 )? Yes, because for c = 13 and n o ≥ 1, Yes, because for c = 2 and n o ≥ 1, 3 n 2 + 4 n + 6 ≥ 2 n 2 3 n 2 + 4 n + 6 ≤ 3 n 2 + 4 n 2 + 6 n 2 = 13 n 2 2 3 n 2 + 4 n + 6 = Ω( n 3 )? 2 3 n 2 + 4 n + 6 = O ( n 3 )? No, because for c = 13 and n o ≥ 1, 3 n 2 + 4 n + 6 < 13 n 3 Yes, because for c = 1 and n o ≥ 13, 3 n 2 + 4 n + 6 ≤ 3 n 2 + 4 n 2 + 6 n 2 = 13 n 2 ≤ 13 n 3 3 3 n 2 + 4 n + 6 = Ω( n )? 3 3 n 2 + 4 n + 6 = O ( n )? Yes, because for c = 2 and n 0 ≥ 100, 3 n 2 + 4 n + 6 > 2 n No, because c · n < 3 n 2 + 4 n + 6 when n > c 15 / 28 16 / 28 Big Theta Examples Exercises Definition T ( n ) is Θ( f ( n )) if there exist constants c 1 > 0 , c 2 > 0 and n 0 ≥ 0 such that c 1 · f ( n ) ≤ T ( n ) ≤ c 2 · f ( n ) for all n ≥ n 0 . 1 Is 3 n + 4 n = O ( n 2 )? (yes or no) 2 Is 2 n + 10 n = Ω( n 3 )? (yes or no) 1 3 n 2 + 4 n + 6 = Θ( n 2 )? 3 Pick a suitable c and n 0 to show that 3 n 2 + 2 n = Ω( n 2 ) Yes, because O and Ω apply 2 3 n 2 + 4 n + 6 = Θ( n 3 )? No, because only O applies 3 3 n 2 + 4 n + 6 = Θ( n )? No, because only Ω applies 17 / 28 18 / 28

Big Oh Addition/Subtraction Big Oh Multiplication Multiplication by a constant does not change the asymptotics Suppose f ( n ) = O ( n 2 ) and g ( n ) = O ( n 2 ). O ( c · f ( n )) → O ( f ( n )) What do we know about g ′ ( n ) = f ( n ) + g ( n )? Ω( c · f ( n )) → Ω( f ( n )) Adding bounding constants shows g ′ ( n ) = O ( n 2 ) Θ( c · f ( n )) → Θ( f ( n )) What do we know about g ′′ ( n ) = f ( n ) − g ( n )? But when both functions in a product are increasing, both are Since bounding constants may not cancel, g ′′ ( n ) = O ( n 2 ) important What about lower bounds? Does g ′ ( n ) = Ω( n 2 ) O ( f ( n )) · O ( g ( n )) → O ( f ( n ) · g ( n )) We know nothing about lower bounds on g ′ and g ′′ because we don’t Ω( f ( n )) · Ω( g ( n )) → Ω( f ( n ) · g ( n )) know about the lower bounds on f and g . Θ( f ( n )) · Θ( g ( n )) → Θ( f ( n ) · g ( n )) 19 / 28 20 / 28 Logarithms Big-Oh notation with multiple variables Upper bounds. T ( m , n ) is O ( f ( m , n )) if there exist constants c > 0 , m 0 ≥ 0 , and n 0 ≥ 0 such that T ( m , n ) ≤ c · f ( m , n ) for all n ≥ n 0 and m ≥ m 0 . Ex. T ( m , n ) = 32 mn 2 + 17 mn + 32 n 3 . It is important to understand deep in your bones what logarithms are ・ T ( m , n ) is both O ( mn 2 + n 3 ) and O ( mn 3 ) . and where they come from. ・ T ( m , n ) is neither O ( n 3 ) nor O ( mn 2 ) . A logarithm is simply an inverse exponential function. Saying b x = y is equivalent to saying that x = log b y . Typical usage. Breadth-first search takes O ( m + n ) time to find the shortest path from s to t in a digraph. Logarithms reflect how many times we can double something until we get to n , or halve something until we get to 1. 17 22 / 28