Algorithmns and Datastructures O-Notation, L Hopital - PowerPoint PPT Presentation

Algorithmns and Datastructures O-Notation, L ’Hopital Albert-Ludwigs-Universität Freiburg Prof. Dr. Rolf Backofen Bioinformatics Group / Department of Computer Science Algorithmns and Datastructures, November 2018

Structure O -Notation Motivation / Definition Examples Ω-Notation Θ-Notation Runtime Summary Limit / Convergence L ’Hôpital / l’Hospital Practical use November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 2 / 56

O -Notation Motivation We are interested in: Example: sorting n 2 Runtime of Minsort “is growing as” Runtime of Heapsort “is growing as” n log n Growth of a function in runtime T ( n ) the role of constants (e.g. 1 ns ) is minor it is enough if relation holds for some n ≥ ... Describe the growth of the function more formally by the means of Landau-Symbols [Wik]): O ( n ) (Big O of n ), Ω( n ) (Omega of n ), Θ( n ) (Theta of n ) November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 4 / 56

O -Notation Definition Big O -Notation: Consider the function: f : N → R , n �→ f ( n ) N : Natural numbers → input size R : Real numbers → runtime Example: f ( n ) = 3 n f ( n ) = 2 n log n f ( n ) = 1 10 n 2 f ( n ) = n 2 +3 n log n − 4 n November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 5 / 56

O -Notation Definition Big O -Notation: Given two functions f and g : f , g : N → R Intuitive: f is Big-O of g ( f is O ( g )) ... if f relative to g does not grow faster than g the growth rate matters, not the absolute values November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 6 / 56

O -Notation Definition Big O -Notation: Informal: f = O ( g ) “=” corresponds to "is" not "is equal to" ... if for some value n 0 for all n ≥ n 0 f ( n ) ≤ C · g ( n ) for a constant C ( f = O ( g ): From a value n 0 for all n ≥ n 0 → f ( n ) ≤ C · g ( n )) Formal: f ∈ O ( g ) Formal: f ∈ O ( g ) O ( g ) = { f : N → R | ∃ n 0 ∈ N , ∃ C > 0 , ∀ n > n 0 : f ( n ) ≤ C · g ( n ) } “set of “for which” “it exists” “for all” “such that” all functions” November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 7 / 56

O -Notation Examples Illustration of the Big O-Notation: g ( n ) f 2 ∈ O ( g ) f 1 ∈ O ( g ) Figure: Runtime of two algorithms f 1 , f 2 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 9 / 56

O -Notation Examples Example: f ( n ) = 5 n +7 , g ( n ) = n ⇒ 5 n +7 ∈ O ( g ) ⇒ f ∈ O ( g ) Intuitive: f ( n ) = 5 n +7 → linear growth Attention f ( n ) ≤ g ( n ) is not guaranteed, better is f ( n ) ≤ C · g ( n ) ∀ n > n 0 . November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 10 / 56

O -Notation Proof We have to proof: ∃ n 0 , ∃ C , ∀ n ≥ n 0 : 5 n +7 ≤ C · n . 5 n +7 ≤ 5 n + n (for n ≥ 7) = 6 n ⇒ n 0 = 7 , C = 6 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 11 / 56

O -Notation Proof Alternate proof: 5 n +7 ≤ 5 n +7 n (for n ≥ 1) = 12 n ⇒ n 0 = 1 , C = 12 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 12 / 56

O -Notation Examples Big O-Notation: We are only interested in the term with the highest-order, the fastest growing summand, the others will be ignored f ( n ) is limited from above by C · g ( n ) Examples: 2 n 2 +7 n − 20 ∈ O ( n 2 ) 2 n 2 +7 n log n − 20 ∈ 7 n log n − 20 ∈ 5 ∈ 2 n 2 +7 n log n + n 3 ∈ November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 13 / 56

O -Notation Examples Harder Example: Polynomes are simple More problematic: combination of complex functions √ x +3ln x ∈ O (??) 2 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 14 / 56

Ω -Notation Definition Omega-Notation: Intuitive : f ∈ Ω( g ), f is growing at least as fast as g So the same as Big-O but with at-least and not at-most Formal: f ∈ Ω( g ) Ω( g ) = { f : N → R | ∃ n 0 ∈ N , ∃ C > 0 , ∀ n > n 0 : f ( n ) ≥ C · g ( n ) } “in O ( n ) we had ≤ ” November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 16 / 56

Ω -Notation Proof Example: Proof of f ( n ) = 5 n +7 ∈ Ω( n ): 5 n +7 ≥ 1 · n (for n ≥ 1) � �� � ���� f ( n ) g ( n ) ⇒ n 0 = 1 , C = 1 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 17 / 56

Ω -Notation Examples Illustration of the Omega-Notation: f 2 ∈ Ω( g ) f 1 ∈ Ω( g ) g ( n ) Figure: Runtime of two algorithms f 1 , f 2 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 18 / 56

Ω -Notation Examples Big Omega-Notation: We are only interested in the term with the highest-order, the fastest growing summand, the others will be ignored f ( n ) is limited from underneath by C · g ( n ) Examples: 2 n 2 +7 n − 20 ∈ Ω( n 2 ) 2 n 2 +7 n log n − 20 ∈ 7 n log n − 20 ∈ 5 ∈ 2 n 2 +7 n log n + n 3 ∈ November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 19 / 56

Θ -Notation Definition Theta-Notation: Intuitive : f is Theta of g ... ... if f is growing as much as g f ∈ Θ( g ), f is growing at the same speed as g Formal: f ∈ Θ( g ) Θ( g ) = O ( g ) ∩ Ω( g ) � �� � Intersection Example: f ( n ) = 5 n +7 , f ( n ) ∈ O ( n ) , f ( n ) ∈ Ω( n ) ⇒ f ( n ) ∈ Θ( n ) Proof for O ( g ) and Ω( g ) look at slides 11 and 17 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 21 / 56

Θ -Notation Graphs number of steps upper bound (runtime (s)) runtime of algorithm lower bound n 0 n f and g have the same “growth” November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 22 / 56

Runtime Landau-Symbol Summary Big O-Notation O ( n ) : f is growing at most as fast as g C · g ( n ) is the upper bound Big Omega-Notation Ω( n ) : f is growing at least as fast as g C · g ( n ) is the lower bound Big Theta-Notation Θ( n ) : f is growing at the same speed as g C 1 · g ( n ) is the lower bound C 2 · g ( n ) is the upper bound November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 24 / 56

Runtime Common Runtimes Table: Common runtime types Runtime Growth f ∈ Θ(1) constant time f ∈ Θ(log n ) = Θ(log k n ) logarithmic time f ∈ Θ( n ) linear time f ∈ Θ( n log n ) n-log-n time (nearly linear) f ∈ Θ( n 2 ) squared time f ∈ Θ( n 3 ) cubic time f ∈ Θ( n k ) polynomial time f ∈ Θ( k n ), f ∈ Θ(2 n ) exponential time November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 25 / 56

So far discussed: Membership in O ( ... ) proofed by hand: Explicit calculation of n 0 and C However: Both hint at limits in calculus November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 27 / 56

O -Notation Limit / Convergence Definition of “Limit” The limit L exists for an infinite sequence f 1 , f 2 , f 3 ,... if for all ε > 0 one n 0 ∈ N exists, such that for all n ≥ n 0 the following holds true: | f n − L | ≤ ε A function f : N → R can be written as a sequence ⇒ lim n → ∞ f n = L The limit is converging: ∀ ε > 0 ∃ n 0 ∈ N ∀ n ≥ n 0 : | f n − L | ≤ ε November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 28 / 56

O -Notation Limit / Convergence Example for the proof of a limit Function f ( n ) = 2+ 1 n with limes lim n → ∞ f ( n ) = 2 “Engineering” solution: use n = ∞ 1 n → ∞ 2+ 1 ∞ = 0 ⇒ lim n → ∞ f ( n ) = lim n = 2 November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 29 / 56

O -Notation Limit / Convergence n → ∞ 2+ 1 Now a more formal proof for lim n = 2 We need to show: for all given ε there is an n 0 such that for all n ≥ n 0 � � � � � 2+ 1 1 � � � � n − 2 � ≤ ε � = � � � � n � E.g.: for ε = 0 . 01 we get 1 n ≤ ε for n ≥ 100 In general � 1 � n 0 = ε Then we get: � � � � 1 � = 1 n ≤ 1 1 � ≤ 1 � � = = ε � � � n n 0 1 1 � ε ε November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 30 / 56

O -Notation Limit / Convergence Let f , g : N → R with an existing limit f ( n ) lim g ( n ) = L n → ∞ Hence the following holds: f ( n ) f ∈ O ( g ) ⇔ lim g ( n ) < ∞ (1) n → ∞ f ( n ) f ∈ Ω( g ) ⇔ g ( n ) > 0 lim (2) n → ∞ f ( n ) f ∈ Θ( g ) ⇔ 0 < lim g ( n ) < ∞ (3) n → ∞ November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 31 / 56

O -Notation Limit / Convergence f ( n ) f ∈ O ( g ) ⇔ lim g ( n ) < ∞ n → ∞ Forward proof ( ⇒ ): def. of O ( n ) f ∈ O ( g ) ⇒ ∃ n 0 , C ∀ n ≥ n 0 : f ( n ) ≤ C · g ( n ) ⇒∃ n 0 , C ∀ n ≥ n 0 : f ( n ) g ( n ) ≤ C f ( n ) ⇒ lim g ( n ) ≤ C n → ∞ November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 32 / 56

O -Notation Limit / Convergence Backward proof ( ⇐ ): f ( n ) lim g ( n ) < ∞ n → ∞ f ( n ) ⇒ lim g ( n ) = C For some C ∈ R (Limit) n → ∞ f ( n ) def. limes ⇒ ∃ n 0 , ∀ n ≥ n 0 : g ( n ) ≤ C + ε ( e . g . ε = 1) ⇒ ∃ n 0 , ∀ n ≥ n 0 : f ( n ) ≤ ( C +1) · g ( n ) � �� � O − notation constant ⇒ f ∈ O ( g ) November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 33 / 56

Limits with L’Hôpital Intuitive: n → ∞ 2+ 1 n = 2+ 1 lim ∞ = 2 With L’Hôpital: Let f , g : N → R If lim n → ∞ f ( n ) = lim n → ∞ g ( n ) = ∞ / 0 f ′ ( n ) f ( n ) ⇒ lim g ( n ) = lim g ′ ( n ) n → ∞ n → ∞ Holy inspiration you need a doctoral degree for that November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 35 / 56