CPSC 320: Intermediate Algorithm Design and Analysis July 18, 2014 - PowerPoint PPT Presentation

CPSC 320: Intermediate Algorithm Design and Analysis July 18, 2014 1

Course Outline Introduction and basic concepts • Asymptotic notation • Greedy algorithms • Graph theory • Amortized analysis • Recursion • Divide-and-conquer algorithms • Randomized algorithms • Dynamic programming algorithms • NP-completeness • 2

Recurrence Relations 3

Recursive Functions Complexity analysis of iterative functions has been covered • How can we analyse the complexity of a recursive function? • Example: • Algorithm MergeSort( 𝐵 , 𝑡 , 𝑓 ) If ( 𝑡 < 𝑓 ) Then 𝑛 ← ⌊ 𝑡 + 𝑓 2⌋ MergeSort( 𝐵 , 𝑡 , 𝑛 ) MergeSort( 𝐵 , 𝑛 + 1 , 𝑓 ) Merge( 𝐵 , 𝑡 , 𝑛 , 𝑓 ) 4

Recurrence Relations The complexity of MergeSort can be computed as: • Θ(1) 𝑜 = 1 𝑜 𝑜 𝑈 𝑜 = 𝑈 + 𝑈 + Θ(𝑜) 𝑜 ≥ 2 2 2 How can we figure out what the complexity is for the function above? • 5

Guess and Test First approach: guess one expression and test if it works • Test usually involves strong induction • Example: prove that 𝑈 𝑜 ≤ 𝑑𝑜 log 𝑜 • Induction step: • 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒𝑜 2 ≤ 2𝑑 𝑜 2 log 𝑜 2 + 𝑒𝑜 ≤ 2𝑑 𝑜 2 log 𝑜 2 + 𝑒𝑜 = 𝑑𝑜 log 𝑜 − 1 + 𝑒𝑜 = 𝑑𝑜 log 𝑜 − 𝑑 − 𝑒 𝑜 ≤ 𝑑𝑜 log 𝑜 if 𝑑 ≥ 𝑒 6

Guess and Test Base case: • 𝑜 = 1 : • 𝑈 1 ≤ 𝑑 ⋅ 1 log 1 1 ≤ 0 doesn’t work! 𝑜 = 2 : • 𝑈 2 ≤ 𝑑 ⋅ 2 log 2 2𝑈 1 + 2𝑒 ≤ 2𝑑 1 + 𝑒 ≤ 𝑑 ∴ 𝑑 ≥ 𝑒 + 1 2 is not enough, since 3 2 = 1 7

Guess and Test Base case (cont.): • 𝑜 = 3 : • 𝑈 3 ≤ 𝑑 ⋅ 3 log 3 2𝑈 1 + 3𝑒 ≤ (3 log 3)𝑑 2 + 3𝑒 3 log 3 ≤ 𝑑 ∴ 𝑑 ≥ 𝑒 + 1 So, for 𝑑 = 𝑒 + 1 and 𝑜 ≥ 𝑜 0 = 2 , 𝑈 𝑜 ∈ 𝑃 𝑜 log 𝑜 • 8

Guess and Test 1 𝑜 = 1 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒𝑜 𝑜 > 1 2 Prove that 𝑈 𝑜 ≤ 𝑑𝑜 • Induction step: • 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒𝑜 2 ≤ 2𝑑 𝑜 2 + 𝑒𝑜 ≤ 2𝑑 𝑜 2 + 𝑒𝑜 = 𝑑𝑜 + 𝑒𝑜 = (𝑑 + 𝑒)𝑜 Prove didn’t reach expected result, disproved • Careful, even though c+d is a constant, we needed the constant to be c only. • 9

Guess and Test 1 𝑜 = 1 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒 𝑜 > 1 2 Prove that 𝑈 𝑜 ≤ 𝑑𝑜 • Induction step: • 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒 2 ≤ 2𝑑 𝑜 2 + 𝑒 ≤ 2𝑑 𝑜 2 + 𝑒 = 𝑑𝑜 + 𝑒 Prove didn’t reach expected result, but we got close • Let’s try a stronger claim: 𝑈 𝑜 ≤ 𝑑𝑜 − 𝑒 ≤ 𝑑𝑜 • 10

Guess and Test 1 𝑜 = 1 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒 𝑜 > 1 2 Prove that 𝑈 𝑜 ≤ 𝑑𝑜 − 𝑒 • Induction step: • 𝑜 𝑈 𝑜 = 2𝑈 + 𝑒 2 ≤ 2 𝑑 𝑜 2 − 𝑒 + 𝑒 ≤ 2𝑑 𝑜 2 − 2𝑒 + 𝑒 = 𝑑𝑜 − 𝑒 Base case, 𝑜 = 1 • 𝑈 1 ≤ 𝑑 ⋅ 1 − 𝑒 1 ≤ 𝑑 − 𝑒 Proved that 𝑑 ≤ 𝑑𝑜 − 𝑒 for 𝑑 = 𝑒 + 1 and 𝑜 ≥ 𝑜 0 = 1 • 11

Recursion Trees 12

Recursion Trees Sometimes guessing isn’t that easy, or that exact • Draw a tree representing all calls to the recursion • Add the “costs” for each row of the tree • Use these sums and the height of the tree to bound the running time • If result is not clear, use substitution to prove a better guess • 13

Examples 𝑜 4 + Θ(𝑜 2 ) 𝑈 𝑜 = 3𝑈 𝑜 ≥ 4 • 𝑒𝑜 2 𝑈 𝑜 Θ(1) 𝑜 < 4 𝑈 𝑜 𝑈 𝑜 𝑈 𝑜 3𝑒𝑜 2 4 4 4 16 𝑜 𝑜 𝑜 𝑜 9𝑒𝑜 2 𝑈 𝑈 𝑈 𝑈 … 16 … 16 … 16 16 256 14

Examples So we have: • log 4 𝑜 𝑈 𝑜 ≤ 𝑒𝑜 2 + 3𝑒𝑜 2 16 + 9𝑒𝑜 2 3 𝑒𝑜 2 256 + ⋯ + 16 2 3 3 3 3 𝑜 2 ≤ 𝑒 16 + + + ⋯ 16 16 ≤ 𝑑𝑜 2 𝑈 𝑜 ∈ 𝑃(𝑜 2 ) 15

Examples 𝑜 𝑈 𝑜 = 3𝑈 2 + Θ(𝑜) 𝑜 ≥ 2 • Θ(1) 𝑜 = 1 First tree level: 𝑒𝑜 • 3 Second tree level: 2 𝑒𝑜 • 2 Third tree level: 3 𝑒𝑜 • 2 Number of levels: log 2 𝑜 • 16

Examples So we have: • log 2 𝑜 𝑈 𝑜 ≤ 𝑒𝑜 + 3𝑒𝑜 + 9𝑒𝑜 + ⋯ + 3 𝑒𝑜 2 4 2 1+lg 𝑜 3 lg 𝑜 3 𝑗 − 1 2 ≤ 𝑒𝑜 = 𝑒𝑜 3 2 2 − 1 𝑗=0 = 3𝑒𝑜 2 lg 3 lg 𝑜 lg 𝑜 = 3𝑒𝑜 3 lg 𝑜 ≤ 2𝑒𝑜 3 3 2 2 2 lg 𝑜 𝑜 = 3𝑒𝑜 lg 3 𝑈 𝑜 ∈ 𝑃(𝑜 lg 3 ) 17

Examples 𝑜 2𝑜 𝑈 𝑜 = 𝑈 3 + 𝑈 + Θ(𝑜) 𝑜 ≥ 3 • 3 Θ(1) 𝑜 < 3 First tree level: 𝑒𝑜 • Second tree level: 𝑒𝑜 • Third tree level: 𝑒𝑜 • Number of levels: log 3 𝑜 on one side, log 3 2 𝑜 on the other side (tree is skewed) • 18

Examples So we have: • 𝑈 𝑜 ≤ 𝑒𝑜 ⋅ log 3 𝑜 2 𝑈 𝑜 ∈ 𝑃(𝑜 log 𝑜) 𝑈 𝑜 ≥ 𝑒𝑜 ⋅ log 3 𝑜 𝑈 𝑜 ∈ Ω(𝑜 log 𝑜) 𝑈 𝑜 ∈ Θ 𝑜 log 𝑜 19

Examples 𝑈 𝑜 = 𝑈 𝑜 − 1 + Θ 1 𝑜 ≥ 1 • Θ 1 𝑜 < 1 First tree level: 𝑒 • Second tree level: 𝑒 • Third tree level: 𝑒 • Number of levels: 𝑜 • Total: 𝑈 𝑜 ≤ 𝑒𝑜 , so 𝑈 𝑜 ∈ 𝑃(𝑜) • 20

Examples So we have: • 𝑈 𝑜 ≤ 𝑒𝑜 ⋅ log 3 𝑜 2 𝑈 𝑜 ∈ 𝑃(𝑜 log 𝑜) 𝑈 𝑜 ≥ 𝑒𝑜 ⋅ log 3 𝑜 𝑈 𝑜 ∈ Ω(𝑜 log 𝑜) 𝑈 𝑜 ∈ Θ 𝑜 log 𝑜 21

Master Theorem 22

Master Theorem Let 𝑏 ≥ 1, 𝑐 > 1 be real constants, 𝑔 𝑜 : ℕ → ℝ + , and 𝑈(𝑜) defined by: • 𝑈 𝑜 = 𝑏𝑈 𝑜 𝑐 + 𝑔(𝑜) 𝑜 ≥ 𝑜 0 Θ(1) 𝑜 < 𝑜 0 where 𝑜 𝑜 𝑜 𝑐 could also be 𝑐 or 𝑐 . Then: • for some 𝜁 > 0 , then 𝑈 𝑜 ∈ Θ 𝑜 log 𝑐 𝑏 . 1. If 𝑔 𝑜 ∈ 𝑃 𝑜 log 𝑐 𝑏−𝜁 2. If 𝑔 𝑜 ∈ Θ 𝑜 log 𝑐 𝑏 log 𝑙 𝑜 for some 𝑙 ≥ 0 , then 𝑈 𝑜 ∈ Θ 𝑜 log 𝑐 𝑏 log 𝑙+1 𝑜 . 𝑜 3. If 𝑔 𝑜 ∈ Ω 𝑜 log 𝑐 𝑏+𝜁 for some 𝜁 > 0 , and 𝑏𝑔 𝑐 < 𝜀𝑔(𝑜) for some 0 < 𝜀 < 1 and all 𝑜 large enough, then 𝑈 𝑜 ∈ Θ 𝑔(𝑜) . 23

Master Theorem Proof We can prove using a recursion tree. • Number of levels: log 𝑐 𝑜 • First level: 𝑔(𝑜) • 𝑜 Second level: 𝑏𝑔 • 𝑐 𝑜 Third level: 𝑏 2 𝑔 • 𝑐 2 𝑜 Close to end: 𝑏 log 𝑐 𝑜−1 𝑔 • 𝑐 log𝑐 𝑜−1 Last level: 𝑏 log 𝑐 𝑜 𝑔 1 • 24

Master Theorem Proof log 𝑐 𝑜−1 𝑜 𝑈 𝑜 = 𝑏 log 𝑐 𝑜 + 𝑏 𝑘 𝑔 𝑐 𝑘 𝑘=0 log 𝑐 𝑜−1 𝑜 = 𝑐 log 𝑐 𝑏 log 𝑐 𝑜 + 𝑏 𝑘 𝑔 𝑐 𝑘 𝑘=0 log 𝑐 𝑜−1 𝑜 = 𝑜 log 𝑐 𝑏 + 𝑏 𝑘 𝑔 𝑐 𝑘 𝑘=0 From this result we can prove all three cases. • 25