Sorting Carola Wenk Slides courtesy of Charles Leiserson with small - PowerPoint PPT Presentation

CS 5633 -- Spring 2008 Sorting Carola Wenk Slides courtesy of Charles Leiserson with small changes by Carola Wenk 2/07/08 CS 5633 Analysis of Algorithms 1

How fast can we sort? All the sorting algorithms we have seen so far are comparison sorts : only use comparisons to determine the relative order of elements. • E.g ., insertion sort, merge sort, quicksort, heapsort. The best worst-case running time that we’ve seen for comparison sorting is O ( n log n ). Is O(nlogn) the best we can do? Decision trees can help us answer this question. 2/07/08 CS 5633 Analysis of Algorithms 2

Decision-tree example Sort 〈 a 1 , a 2 , …, a n 〉 a 1 : a 2 a 1 : a 2 ≥ < a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 ≥ < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. •The left subtree shows subsequent comparisons if a i < a j . •The right subtree shows subsequent comparisons if a i ≥ a j . 2/07/08 CS 5633 Analysis of Algorithms 3

Decision-tree example Sort 〈 a 1 , a 2 , a 3 〉 a 1 : a 2 a 1 : a 2 ≥ < = 〈 9, 4, 6 〉: a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 ≥ < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. •The left subtree shows subsequent comparisons if a i < a j . •The right subtree shows subsequent comparisons if a i ≥ a j . 2/07/08 CS 5633 Analysis of Algorithms 4

Decision-tree example Sort 〈 a 1 , a 2 , a 3 〉 a 1 : a 2 9 ≥ 4 a 1 : a 2 < = 〈 9, 4, 6 〉: a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 ≥ < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. •The left subtree shows subsequent comparisons if a i < a j . •The right subtree shows subsequent comparisons if a i ≥ a j . 2/07/08 CS 5633 Analysis of Algorithms 5

Decision-tree example Sort 〈 a 1 , a 2 , a 3 〉 a 1 : a 2 a 1 : a 2 ≥ < = 〈 9, 4, 6 〉: a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 9 ≥ 6 < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. •The left subtree shows subsequent comparisons if a i < a j . •The right subtree shows subsequent comparisons if a i ≥ a j . 2/07/08 CS 5633 Analysis of Algorithms 6

Decision-tree example Sort 〈 a 1 , a 2 , a 3 〉 a 1 : a 2 a 1 : a 2 ≥ < = 〈 9, 4, 6 〉: a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 ≥ < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < ≥ ≥ 4 < 6 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 1 a 2 a 3 a 3 a 2 a 1 Each internal node is labeled a i : a j for i , j ∈ {1, 2,…, n }. •The left subtree shows subsequent comparisons if a i < a j . •The right subtree shows subsequent comparisons if a i ≥ a j . 2/07/08 CS 5633 Analysis of Algorithms 7

Decision-tree example Sort 〈 a 1 , a 2 , a 3 〉 a 1 : a 2 a 1 : a 2 ≥ < = 〈 9, 4, 6 〉: a 2 : a 3 a 1 : a 3 a 2 : a 3 a 1 : a 3 ≥ < ≥ < a 1 a 2 a 3 a 2 a 1 a 3 a 1 a 2 a 3 a 1 : a 3 a 2 a 1 a 3 a 2 : a 3 a 1 : a 3 a 2 : a 3 < < ≥ ≥ a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 a 1 a 3 a 2 a 3 a 1 a 2 a 2 a 3 a 1 a 3 a 2 a 1 4 < 6 ≤ 9 Each leaf contains a permutation 〈π(1) , π(2) ,…, π ( n ) 〉 to indicate that the ordering a π (1) ≤ a π (2) ≤ L ≤ a π (n) has been established. 2/07/08 CS 5633 Analysis of Algorithms 8

Decision-tree model A decision tree models the execution of any comparison sorting algorithm: • One tree per input size n . • The tree contains all possible comparisons (= if-branches) that could be executed for any input of size n . • The tree contains all comparisons along all possible instruction traces (= control flows) for all inputs of size n . • For one input, only one path to a leaf is executed. • Running time = length of the path taken. • Worst-case running time = height of tree. 2/07/08 CS 5633 Analysis of Algorithms 9

Lower bound for comparison sorting Theorem. Any decision tree that can sort n elements must have height Ω ( n log n ). Proof. The tree must contain ≥ n ! leaves, since there are n ! possible permutations. A height- h binary tree has ≤ 2 h leaves. Thus, n ! ≤ 2 h . ∴ h ≥ log( n !) (log is mono. increasing) ≥ log (( n / e ) n ) (Stirling’s formula) = n log n – n log e ⇒ h ∈ Ω ( n log n ) . 2/07/08 CS 5633 Analysis of Algorithms 10

Lower bound for comparison sorting Corollary. Heapsort and merge sort are asymptotically optimal comparison sorting algorithms. 2/07/08 CS 5633 Analysis of Algorithms 11

Sorting in linear time Counting sort: No comparisons between elements. • Input : A [1 . . n ], where A [ j ] ∈ {1, 2, …, k } . • Output : B [1 . . n ], sorted. • Auxiliary storage : C [1 . . k ] . 2/07/08 CS 5633 Analysis of Algorithms 12

Counting sort for i ← 1 to k 1. do C [ i ] ← 0 for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 for i ← 2 to k 3. ⊳ C [ i ] = |{key ≤ i }| do C [ i ] ← C [ i ] + C [ i –1] for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/07/08 CS 5633 Analysis of Algorithms 13

Counting-sort example 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 4 1 3 4 3 B : 2/07/08 CS 5633 Analysis of Algorithms 14

Loop 1 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 0 0 0 0 4 1 3 4 3 0 0 0 0 B : for i ← 1 to k 1. do C [ i ] ← 0 2/07/08 CS 5633 Analysis of Algorithms 15

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 0 0 0 1 4 1 3 4 3 0 0 0 1 B : for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 2/07/08 CS 5633 Analysis of Algorithms 16

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 0 1 4 1 3 4 3 1 0 0 1 B : for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 2/07/08 CS 5633 Analysis of Algorithms 17

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 1 1 4 1 3 4 3 1 0 1 1 B : for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 2/07/08 CS 5633 Analysis of Algorithms 18

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 1 2 4 1 3 4 3 1 0 1 2 B : for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 2/07/08 CS 5633 Analysis of Algorithms 19

Loop 2 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 4 1 3 4 3 1 0 2 2 B : for j ← 1 to n 2. ⊳ C [ i ] = |{key = i }| do C [ A [ j ]] ← C [ A [ j ]] + 1 2/07/08 CS 5633 Analysis of Algorithms 20

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 4 1 3 4 3 1 0 2 2 B : C' : 1 1 2 2 1 1 2 2 for i ← 2 to k 3. ⊳ C [ i ] = |{key ≤ i }| do C [ i ] ← C [ i ] + C [ i –1] 2/07/08 CS 5633 Analysis of Algorithms 21

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 4 1 3 4 3 1 0 2 2 B : C' : 1 1 3 2 1 1 3 2 for i ← 2 to k 3. ⊳ C [ i ] = |{key ≤ i }| do C [ i ] ← C [ i ] + C [ i –1] 2/07/08 CS 5633 Analysis of Algorithms 22

Loop 3 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 0 2 2 4 1 3 4 3 1 0 2 2 B : C' : 1 1 3 5 1 1 3 5 for i ← 2 to k 3. ⊳ C [ i ] = |{key ≤ i }| do C [ i ] ← C [ i ] + C [ i –1] 2/07/08 CS 5633 Analysis of Algorithms 23

Loop 4 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 1 3 5 4 1 3 4 3 1 1 3 5 B : 3 C' : 1 1 3 5 3 1 1 3 5 for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/07/08 CS 5633 Analysis of Algorithms 24

Loop 4 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 1 3 5 4 1 3 4 3 1 1 3 5 B : 3 C' : 1 1 2 5 3 1 1 2 5 for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/07/08 CS 5633 Analysis of Algorithms 25

Loop 4 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 1 2 5 4 1 3 4 3 1 1 2 5 B : 3 4 C' : 1 1 2 5 3 4 1 1 2 5 for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/07/08 CS 5633 Analysis of Algorithms 26

Loop 4 1 2 3 4 5 1 2 3 4 A : 4 1 3 4 3 C : 1 1 2 5 4 1 3 4 3 1 1 2 5 B : 3 4 C' : 1 1 2 4 3 4 1 1 2 4 for j ← n downto 1 4. do B [ C [ A [ j ]]] ← A[ j ] C [ A [ j ]] ← C [ A [ j ]] – 1 2/07/08 CS 5633 Analysis of Algorithms 27