Computational Complexity 15-150 1
Desirable Properties of Programs • What is the most desirable property of a program? • Correctness • Nobody cares about incorrect programs. • Assuming you have a correct program, what is the next desirable property? • Efficiency – Programs should not be using unnecessary resources • Total computational effort (time/steps etc.) • Memory • Communication 2
Why do we care? • Suppose you have two Programs P1 and P2 for sorting n integers • Assume when n=1000, • P1 takes 10,000 units of time • P2 takes 1,000,000 units of time • Which is preferable? • Obviously P1. • How do you find out? We obviously do NOT really want to run these two programs to find out P2 takes too much time. • We need a simple but reasonably accurate way to find this out! 3
Why do we care? • Less running time -> Less waiting for results • Less running time -> Less power consumption -> Lower cost • Less power consumption -> Less heat produced -> Less cooling -> Lower cost 4
How? Big Picture • Model the effort that needs to be expended when running a program • Work (models overall cost) • Model the opportunities for (ideal) parallel execution • Span (models time waiting for results) 5
How? Big Picture 6
What do we mean by “Model”? • What do we model? • Actual time? • Depends on • Actual hardware, (CPU Speed, Cache Size, etc.) Compiler, optimizer etc. • • Compiled vs Interpreted execution mode. • Way too complicated to model and measure • Instead • Model the algorithm instead of the program. • Model some other proxy for actual effort. • For example, the count of most frequently executed operation • Model accurately enough so that algorithms can be compared. • We are NOT really concerned about all details. 7
Some elaborations • Model the algorithm instead of the program. • This is usually sufficient. • A very efficient implementation of an otherwise inefficient algorithm is not a good idea • Unless your inputs are always expected to be very small. • Model some other proxy for actual effort. • Number of steps, • Number of important operations • Number of instructions executed? • Model accurately enough so that algorithms can be compared. • How does work or span vary as input size (not value) increases? • For example, if my input doubles in size, how does my work increase? • Doubles the first effort? • Square of the first effort? 8
Example How many steps are needed for evalution as a function of the value of n ? (This is really wrong but convenient – bear with me.) 𝑋 𝑜 = Number of steps executed when argument has value 𝑜 9
Example Do you see a pattern here? 10
Example Do you see a pattern here? We always compute for • argument 1 less We have 3 additional steps. • Except for the base case! (1 • step there) 11
Example This is called a recurrence. • Reflects the structure of the code • NOT a closed form -- need to repeatedly expand • With some kindergarten algebra, we can easily find a closed form 12
Example n steps Repeated Expansion 13
Example 𝑋 !"# 𝑜 = 3𝑜 + 1 Case Definition Solution 𝑜 = 0 1 1 𝑋 !"# 𝑜 = 3 + 𝑋 !"# (𝑜 − 1) 𝑜 > 0 = 3 + 3 𝑜 − 1 + 1 3𝑜 + 1 = 3𝑜 + 3 − 3 + 1 = 3𝑜 + 1 14
Another Example What is exp’ computing? 𝑓𝑦𝑞 ! 𝑜 = 2 " What is the cost of square? 2 (body and *) 15
Another Example What is the recurrence for 𝑋 !"#$ (𝑜) ? ( 𝑜 is again the value of the argument) How many cases should we consider? 3 Why 5? Function body, mod, compare =, if, div Function body, mod, compare =, if, -, * Why 6? 16
Another Example Why? Why? 17
Another Example In general we do really care about the constants (1, 7 or 13) • They do not impact the growth rates • They just move the function (cost vs n) up/down the y axis. • We abstract away 18
Another Example Do you see another simplification? For odd 𝑜 , 𝑜 − 1 𝑒𝑗𝑤 2 = 𝑜 𝑒𝑗𝑤 2 where you can pick 𝑙 = max(𝑙 $ , 𝑙 % ) 19
Another Example Again with some kindergarten algebra, we can easily find a closed form 1 + log % 𝑜 steps 𝑋 !"# ! 𝑜 = 𝑙 log % 𝑜 + 1 + 𝑙 & = 𝑙 log % 𝑜 + 𝑙 + 𝑙 & = 𝑙 log % 𝑜 + 𝑙′ For modeling purposes, it is convenient to assume that 𝑜 = 2 ' for some 𝑙 ≥ 0 20
Let’s Compare the Two Algorithms 350 300 250 200 150 100 50 0 10 20 30 40 50 60 70 80 90 100 3n+1 13log_2(n)+15 For small n, the first algorithm seems to be better As n increases, the second algorithm, is much better. We are making one very important but incorrect assumption? Can you spot it? Let me know( J ) • 21
Comparing Algorithms • We really compare the functions’ growth rate. • Constant factors do not matter • Lower order terms do not matter • We use the big-O notation. • We say things like • 𝑋 𝑜 = 3𝑜 + 1 = 𝑃(𝑜) – Function grows linearly • 𝑋 𝑜 = 𝑙 log % 𝑜 + 𝑙 $ = 𝑃(log 𝑜) • Function grows logarithmically • The base of the logarithm does not matter (Why?) • log 𝑜 grows much slower than 𝑜 regardless of the constant multipliers. 22
The big-O notation • We say a function 𝑔 𝑜 = 𝑃( 𝑜 ) iff there are constants 𝑑 > 0 and 𝑜 ' ≥ 0 , such that for all 𝑜 ≥ 𝑜 ' , 𝑑 𝑜 ≥ 𝑔 𝑜 . • You can also see uses as 𝑔 𝑜 ∈ 𝑃( 𝑜 ) • The set of all functions that grow at most as 𝑜 . • We say 𝑑 𝑜 is an upper bound for 𝑔(𝑜) . 23
The big-O notation 24
What is the argument of the work model? • The first examples took n to be the value. • This is technically wrong! 25
The big-O notation • 1000 𝑜 ! = O n ! • 𝑜 ! + 1000000𝑜 = 𝑃 𝑜 ! • 5𝑜 log ! 𝑜 = 𝑃(𝑜 log 𝑜) • 5𝑜 log ! 𝑜 "#$ = 𝑃(𝑜 log 𝑜) (Why?) • 5𝑜 = 𝑃(𝑜 % ) (But this is not a very useful statement) • Why? • We never say things like: • 3𝑜 = 𝑃 5𝑜 Why? • 5𝑜 % + 7 = 𝑃(𝑜 + 7) Why? • 𝑃 𝑜 = 5𝑜 + 7 • 𝑃 is always on the right side of =/∈ 26
Another Example – List Reversal We also need to know the complexity of the @ function 27
Complexity of the Append What does 𝑜 represent? The length/size of the first list. The size of the second list is not important. (Why?) This is indeed the correct way to model work (and span). Work and span should be expressed as a function of the size of the input • Not as a function of the input!! (Remember my earlier comment) • 𝑋 @ 𝑜 = 𝑏 ; 𝑜 + 𝑏 < = 𝑃(𝑜) 28
Complexity of rev Substituting 𝑋 @ 𝑜 = 𝑏 " 𝑜 + 𝑏 # and combining constants we get 29
Complexity of rev Lower order terms $%& 𝑜 = 𝑏 " ( − 𝑏 " 2 𝑜 ' + (𝑐 " 𝑋 2 )n + 𝑐 # = 𝑃(𝑜 ' ) 30
Complexity of trev You only need to know the complexity of trevh 𝑋 ABCDE 𝑜 = 𝑑 ; 𝑜 + 𝑑 < = 𝑃(𝑜) Tail recursive reverse is substantially more efficient. 31
Summary • We need to model the efficiency of our algorithms. • Complexity models are follow to the structure of the algorithm. • Accounting for the most important operations are usually sufficient. • For recursively expressed algorithms models lead to recurrences. • (Simple) Recurrences can be solved by simple algebraic expansions. • Models can be expressed with the big-O notation. 32
Recommend
More recommend
