Sorting Carola Wenk Slides courtesy of Charles Leiserson with small - PowerPoint PPT Presentation

CS 3343 – Fall 2010 Sorting Carola Wenk Slides courtesy of Charles Leiserson with small changes by Carola Wenk 10/7/10 1 CS 3343 Analysis of Algorithms

How fast can we sort? All the sorting algorithms we have seen so far are comparison sorts : only use comparisons to determine the relative order of elements. • E.g ., insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we’ve seen for comparison sorting is O ( n log n ). Is O(nlogn) the best we can do? Decision trees can help us answer this question. 10/7/10 2 CS 3343 Analysis of Algorithms

Decision-tree model A decision tree models the execution of any comparison sorting algorithm: • One tree per input size n . • The tree contains all possible comparisons (= if-branches) that could be executed for any input of size n . • The tree contains all comparisons along all possible instruction traces (= control flows) for all inputs of size n . • For one input, only one path to a leaf is executed. • Running time = length of the path taken. • Worst-case running time = height of tree. 10/7/10 3 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 4 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 5 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 9 ≥ 4 < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 6 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j i < 9 ≥ 6 ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 7 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < ≥ 4 < 6 ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 8 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 3 2 1 4 < 6 ≤ 9 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 10/7/10 9 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a 1 : a 3 a 1 a 2 a 3 j ≥ i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 j i a 1 : a 3 a 2 : a 3 1 2 3 2 1 3 i j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 3 2 1 4 < 6 ≤ 9 Each leaf contains a permutation 〈π(1) , π(2) ,…, π ( n ) 〉 to indicate that the ordering a π (1) ≤ a π (2) ≤ ... ≤ a π (n) has been established. 10/7/10 10 CS 3343 Analysis of Algorithms

Decision-tree model A decision tree models the execution of any comparison sorting algorithm: • One tree per input size n . • The tree contains all possible comparisons (= if-branches) that could be executed for any input of size n . • The tree contains all comparisons along all possible instruction traces (= control flows) for all inputs of size n . • For one input, only one path to a leaf is executed. • Running time = length of the path taken. • Worst-case running time = height of tree. 10/7/10 11 CS 3343 Analysis of Algorithms

Lower bound for comparison sorting Theorem. Any decision tree that can sort n elements must have height Ω ( n log n ). Proof. The tree must contain ≥ n ! leaves, since there are n ! possible permutations. A height- h binary tree has ≤ 2 h leaves. Thus, n ! ≤ 2 h . ∴ h ≥ log( n !) (log is mono. increasing) ≥ log (( n /2) n/ 2 ) = n/ 2 log n/ 2 ⇒ h ∈ Ω ( n log n ). 10/7/10 12 CS 3343 Analysis of Algorithms

Lower bound for comparison sorting Corollary. Heapsort and merge sort are asymptotically optimal comparison sorting algorithms. 10/7/10 13 CS 3343 Analysis of Algorithms

Sorting in linear time Counting sort: No comparisons between elements. • Input : A [1 . . n ], where A [ j ] ∈ {1, 2, …, k } . • Output : B [1 . . n ], sorted. • Auxiliary storage : C [1 . . k ]. 10/7/10 14 CS 3343 Analysis of Algorithms

Counting sort for i ← 1 to k 1. do C [ i ] ← 0 for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| for i ← 2 to k 3. do C [ i ] ← C [ i ] + C [ i –1] ? C [ i ] = |{key ≤ i }| for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 10/7/10 15 CS 3343 Analysis of Algorithms

Counting-sort example 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : B : 10/7/10 16 CS 3343 Analysis of Algorithms

Loop 1 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 0 0 0 0 B : for i ← 1 to k 1. do C [ i ] ← 0 10/7/10 17 CS 3343 Analysis of Algorithms

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 0 0 0 1 B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| 10/7/10 18 CS 3343 Analysis of Algorithms

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 0 1 B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| 10/7/10 19 CS 3343 Analysis of Algorithms

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 1 1 B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| 10/7/10 20 CS 3343 Analysis of Algorithms

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 1 2 B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| 10/7/10 21 CS 3343 Analysis of Algorithms

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 ? C [ i ] = |{key = i }| 10/7/10 22 CS 3343 Analysis of Algorithms

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 B : C' : 1 1 2 2 for i ← 2 to k 3. do C [ i ] ← C [ i ] + C [ i –1] ? C [ i ] = |{key ≤ i }| 10/7/10 23 CS 3343 Analysis of Algorithms

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 B : C' : 1 1 3 2 for i ← 2 to k 3. do C [ i ] ← C [ i ] + C [ i –1] ? C [ i ] = |{key ≤ i }| 10/7/10 24 CS 3343 Analysis of Algorithms

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 B : C' : 1 1 3 5 for i ← 2 to k 3. do C [ i ] ← C [ i ] + C [ i –1] ? C [ i ] = |{key ≤ i }| 10/7/10 25 CS 3343 Analysis of Algorithms