How fast can we sort? All the sorting algorithms we have seen so far - PowerPoint PPT Presentation

CS 3343 -- Spring 2009 How fast can we sort? All the sorting algorithms we have seen so far are comparison sorts : only use comparisons to determine the relative order of elements. • E.g ., insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we’ve Sorting seen for comparison sorting is O ( n log n ). Carola Wenk Is O(nlogn) the best we can do? Slides courtesy of Charles Leiserson with small Decision trees can help us answer this question. changes by Carola Wenk 2/24/09 CS 3343 Analysis of Algorithms 1 2/24/09 CS 3343 Analysis of Algorithms 2 Decision-tree model Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 A decision tree models the execution of any a 1 a 2 a 3 insert a 2 comparison sorting algorithm: i j a 1 : a 2 a 1 : a 2 ≥ < insert a 3 • One tree per input size n . a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 • The tree contains all possible comparisons (= if-branches) i j ≥ < ≥ < i j a 1 a 2 a 3 that could be executed for any input of size n . a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 i j a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 • The tree contains all comparisons along all possible i j < < ≥ instruction traces (= control flows) for all inputs of size n . ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 • For one input, only one path to a leaf is executed. a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 • Running time = length of the path taken. Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • Worst-case running time = height of tree. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 2/24/09 CS 3343 Analysis of Algorithms 3 2/24/09 CS 3343 Analysis of Algorithms 4 1

Decision-tree for insertion sort Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 a 1 a 2 a 3 insert a 2 insert a 2 j j i i a 1 : a 2 a 1 : a 2 a 1 : a 2 a 1 : a 2 9 ≥ 4 ≥ < < insert a 3 insert a 3 a 2 a 1 a 3 a 2 a 1 a 3 insert a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 i j i j ≥ ≥ < ≥ < ≥ < < i j i j a 1 a 2 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j i j a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 i j i j < < ≥ < < ≥ ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 2/24/09 CS 3343 Analysis of Algorithms 5 2/24/09 CS 3343 Analysis of Algorithms 6 Decision-tree for insertion sort Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 a 1 a 2 a 3 insert a 2 i j i j a 1 : a 2 a 1 : a 2 a 1 : a 2 a 1 : a 2 ≥ ≥ < < insert a 3 insert a 3 a 2 a 1 a 3 a 2 a 1 a 3 insert a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 i j i j ≥ < ≥ 9 ≥ 6 < ≥ < < i j i j a 1 a 2 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 i j a 2 a 1 a 3 a 2 : a 3 a 1 a 2 a 3 a 1 : a 3 i j a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 i j i j < < ≥ < ≥ 4 < 6 ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 2/24/09 CS 3343 Analysis of Algorithms 7 2/24/09 CS 3343 Analysis of Algorithms 8 2

Decision-tree for insertion sort Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 a 1 a 2 a 3 insert a 2 insert a 2 j j i i a 1 : a 2 a 1 : a 2 a 1 : a 2 a 1 : a 2 ≥ ≥ < < insert a 3 insert a 3 a 2 a 1 a 3 a 2 a 1 a 3 insert a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 a 2 : a 3 a 1 : a 3 i j i j ≥ ≥ < ≥ < ≥ < < i j i j a 1 a 2 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j i j a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 i j i j < < ≥ < < ≥ ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 4 < 6 ≤ 9 4 < 6 ≤ 9 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. Each leaf contains a permutation 〈π(1) , π(2) ,…, π ( n ) 〉 to indicate • The left subtree shows subsequent comparisons if a i < a j . that the ordering a π (1) ≤ a π (2) ≤ ... ≤ a π (n) has been established. • The right subtree shows subsequent comparisons if a i ≥ a j . 2/24/09 CS 3343 Analysis of Algorithms 9 2/24/09 CS 3343 Analysis of Algorithms 10 Lower bound for Decision-tree model comparison sorting A decision tree models the execution of any Theorem. Any decision tree that can sort n comparison sorting algorithm: elements must have height Ω ( n log n ). • One tree per input size n . Proof. The tree must contain ≥ n ! leaves, since • The tree contains all possible comparisons (= if-branches) that could be executed for any input of size n . there are n ! possible permutations. A height- h binary tree has ≤ 2 h leaves. Thus, n ! ≤ 2 h . • The tree contains all comparisons along all possible instruction traces (= control flows) for all inputs of size n . ∴ h ≥ log( n !) (log is mono. increasing) • For one input, only one path to a leaf is executed. ≥ log (( n /2) n/ 2 ) • Running time = length of the path taken. = n/ 2 log n/ 2 • Worst-case running time = height of tree. ⇒ h ∈ Ω ( n log n ) . 2/24/09 CS 3343 Analysis of Algorithms 11 2/24/09 CS 3343 Analysis of Algorithms 12 3

Lower bound for comparison Sorting in linear time sorting Counting sort: No comparisons between elements. Corollary. Heapsort and merge sort are • Input : A [1 . . n ], where A [ j ] ∈ {1, 2, …, k } . asymptotically optimal comparison sorting • Output : B [1 . . n ], sorted. algorithms. • Auxiliary storage : C [1 . . k ]. 2/24/09 CS 3343 Analysis of Algorithms 13 2/24/09 CS 3343 Analysis of Algorithms 14 Counting sort Counting-sort example for i ← 1 to k 1. 1 2 3 4 5 1 2 3 4 do C [ i ] ← 0 A : 4 1 3 4 3 C : 4 1 3 4 3 for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 for i ← 2 to k 3. B : do C [ i ] ← C [ i ] + C [ i –1] ⊳ C [ i ] = |{key ≤ i }| for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/24/09 CS 3343 Analysis of Algorithms 15 2/24/09 CS 3343 Analysis of Algorithms 16 4

Loop 1 Loop 2 1 2 3 4 5 1 2 3 4 5 1 2 3 4 1 2 3 4 A : 4 1 3 4 3 A : 4 1 3 4 3 C : 0 0 0 0 C : 0 0 0 1 4 1 3 4 3 0 0 0 0 4 1 3 4 3 0 0 0 1 B : B : for i ← 1 to k for j ← 1 to n 1. 2. ⊳ C [ i ] = |{key = i }| do C [ i ] ← 0 do C [ A [ j ]] ← C [ A [ j ]] + 1 2/24/09 CS 3343 Analysis of Algorithms 17 2/24/09 CS 3343 Analysis of Algorithms 18 Loop 2 Loop 2 1 2 3 4 5 1 2 3 4 5 1 2 3 4 1 2 3 4 A : 4 1 3 4 3 A : 4 1 3 4 3 C : 1 0 0 1 C : 1 0 1 1 4 1 3 4 3 4 1 3 4 3 1 0 0 1 1 0 1 1 B : B : for j ← 1 to n for j ← 1 to n 2. 2. ⊳ C [ i ] = |{key = i }| ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 do C [ A [ j ]] ← C [ A [ j ]] + 1 2/24/09 CS 3343 Analysis of Algorithms 19 2/24/09 CS 3343 Analysis of Algorithms 20 5