Sorting Carola Wenk Slides courtesy of Charles Leiserson with small - PowerPoint PPT Presentation

CS 3343 – Fall 2011 Sorting Carola Wenk Slides courtesy of Charles Leiserson with small y changes by Carola Wenk 9/29/11 1 CS 3343 Analysis of Algorithms

How fast can we sort? How fast can we sort? All the sorting algorithms we have seen so far are comparison sorts : only use comparisons to determine the relative order of elements. • E.g ., insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we’ve seen for comparison sorting is O ( n log n ). Is O(nlogn) the best we can do? Decision trees can help us answer this question. h l thi ti D i i 9/29/11 2 CS 3343 Analysis of Algorithms

Decision-tree model Decision tree model A decision tree models the execution of any comparison sorting algorithm: • One tree per input size n . • The tree contains all possible comparisons (= if-branches) Th i ibl i ( if b h ) ll that could be executed for any input of size n . • The tree contains all comparisons along all possible The tree contains all comparisons along all possible instruction traces (= control flows) for all inputs of size n . • For one input, only one path to a leaf is executed. p y p • Running time = length of the path taken. • Worst-case running time = height of tree. 9/29/11 3 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 4 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 5 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 9 ≥ 4 < insert a 3 a 2 a 1 a 3 insert a 3 a 2 : a 3 a : a a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 6 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 j i < ≥ 9 ≥ 6 < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 7 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < ≥ 4 < 6 ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 8 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 4 < 6 ≤ 9 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. • The left subtree shows subsequent comparisons if a i < a j . The left subtree shows subsequent comparisons if a i < a j . • The right subtree shows subsequent comparisons if a i ≥ a j . 9/29/11 9 CS 3343 Analysis of Algorithms

Decision-tree for insertion sort Decision tree for insertion sort Sort 〈 a 1 , a 2 , a 3 〉 = <9,4,6> a 1 a 2 a 3 insert a 2 1 2 3 2 j i a 1 : a 2 ≥ < insert a 3 a 2 a 1 a 3 insert a 3 a : a a 2 : a 3 a : a a 1 : a 3 a a a a 1 a 2 a 3 ≥ j i < ≥ < j i a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 2 a 1 a 3 i j a 1 : a 3 a 1 : a 3 a 2 : a 3 a 2 : a 3 1 2 3 2 1 3 i i j j < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 1 3 2 3 1 2 2 3 1 3 2 1 4 < 6 ≤ 9 Each leaf contains a permutation 〈π(1) , π(2) ,…, π ( n ) 〉 to indicate that the ordering a π (1) ≤ a π (2) ≤ ... ≤ a π (n) has been established. 9/29/11 10 CS 3343 Analysis of Algorithms

Lower bound for comparison sorting i i Theorem Any decision tree that can sort n Theorem. Any decision tree that can sort n elements must have height Ω ( n log n ). Proof. The tree must contain ≥ n ! leaves, since t i ≥ ! l f Th t t i P there are n ! possible permutations. A height- h binary tree has ≤ 2 h leaves Thus n ! ≤ 2 h binary tree has ≤ 2 h leaves. Thus, n ! ≤ 2 h . ∴ h ≥ log( n !) (log is mono. increasing) ≥ l ≥ log (( n /2) n/ 2 ) (( /2) n/ 2 ) = n/ 2 log n/ 2 ⇒ h ∈ Ω ( n log n ) . Ω ( l ) h 9/29/11 11 CS 3343 Analysis of Algorithms

Lower bound for comparison sorting i Corollary. Heapsort and merge sort are asymptotically optimal comparison sorting algorithms. 9/29/11 12 CS 3343 Analysis of Algorithms

Sorting in linear time Sorting in linear time Counting sort: No comparisons between elements Counting sort: No comparisons between elements. • Input : A [1 . . n ], where A [ j ] ∈ {1, 2, …, k } . • Output : B [1 . . n ], sorted. [1 ] d O • Auxiliary storage : C [1 . . k ] . 9/29/11 13 CS 3343 Analysis of Algorithms

Counting sort Counting sort 1. for i ← 1 to k for i ← 1 to k 1. do C [ i ] ← 0 2. for j ← 1 to n for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 C [ i ] = |{key = i }| 3. for i ← 2 to k for i ← 2 to k 3. do C [ i ] ← C [ i ] + C [ i –1] C [ i ] = |{key ≤ i }| 4. for j ← n downto 1 for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 C [ A [ j ]] ← C [ A [ j ]] 1 9/29/11 14 CS 3343 Analysis of Algorithms

15 4 4 3 3 Counting sort example Counting-sort example 2 2 1 1 CS 3343 Analysis of Algorithms C : 3 5 5 4 4 4 3 3 3 1 2 2 4 1 1 9/29/11 A : B : B :

16 0 4 4 0 3 3 0 2 2 0 1 1 CS 3343 Analysis of Algorithms C : 3 5 5 4 4 4 Loop 1 Loop 1 do C [ i ] ← 0 for i ← 1 to k 3 3 3 1 2 2 4 1 1 9/29/11 A : B : B : 1.

Loop 2 Loop 2 1 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 A : 4 1 3 4 3 C : 0 0 0 1 B : B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 C [ i ] = |{key = i }| 9/29/11 17 CS 3343 Analysis of Algorithms

Loop 2 Loop 2 1 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 A : 4 1 3 4 3 C : 1 0 0 1 B : B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 C [ i ] = |{key = i }| 9/29/11 18 CS 3343 Analysis of Algorithms

Loop 2 Loop 2 1 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 A : 4 1 3 4 3 C : 1 0 1 1 B : B : for j ← 1 to n 2. do C [ A [ j ]] ← C [ A [ j ]] + 1 C [ i ] = |{key = i }| 9/29/11 19 CS 3343 Analysis of Algorithms