Backtracking and Memoization Lecture 12 Tuesday, February 26, 2019 - PowerPoint PPT Presentation

Algorithms & Models of Computation CS/ECE 374, Spring 2019 Backtracking and Memoization Lecture 12 Tuesday, February 26, 2019 L A T EXed: December 27, 2018 08:25 Chan, Har-Peled, Hassanieh (UIUC) CS374 1 Spring 2019 1 / 22

Recursion Reduction: Reduce one problem to another Recursion A special case of reduction reduce problem to a smaller instance of itself 1 self-reduction 2 Problem instance of size n is reduced to one or more instances 1 of size n − 1 or less. For termination, problem instances of small size are solved by 2 some other method as base cases . Chan, Har-Peled, Hassanieh (UIUC) CS374 2 Spring 2019 2 / 22

Recursion in Algorithm Design Tail Recursion : problem reduced to a single recursive call after 1 some work. Easy to convert algorithm into iterative or greedy algorithms. Examples: Interval scheduling, MST algorithms, etc. Divide and Conquer : Problem reduced to multiple 2 independent sub-problems that are solved separately. Conquer step puts together solution for bigger problem. Examples: Closest pair, deterministic median selection, quick sort. Backtracking : Refinement of brute force search. Build solution 3 incrementally by invoking recursion to try all possibilities for the decision in each step. Dynamic Programming : problem reduced to multiple 4 (typically) dependent or overlapping sub-problems. Use memoization to avoid recomputation of common solutions leading to iterative bottom-up algorithm. Chan, Har-Peled, Hassanieh (UIUC) CS374 3 Spring 2019 3 / 22

Part I Brute Force Search, Recursion and Backtracking Chan, Har-Peled, Hassanieh (UIUC) CS374 4 Spring 2019 4 / 22

Maximum Independent Set in a Graph Definition Given undirected graph G = ( V , E ) a subset of nodes S ⊆ V is an independent set (also called a stable set) if for there are no edges between nodes in S . That is, if u , v ∈ S then ( u , v ) �∈ E . B C A F E D Some independent sets in graph above: { D } , { A , C } , { B , E , F } Chan, Har-Peled, Hassanieh (UIUC) CS374 5 Spring 2019 5 / 22

Maximum Independent Set Problem Input Graph G = ( V , E ) Goal Find maximum sized independent set in G B C A F E D Chan, Har-Peled, Hassanieh (UIUC) CS374 6 Spring 2019 6 / 22

Maximum Weight Independent Set Problem Input Graph G = ( V , E ) , weights w ( v ) ≥ 0 for v ∈ V Goal Find maximum weight independent set in G B C A F E D Chan, Har-Peled, Hassanieh (UIUC) CS374 7 Spring 2019 7 / 22

Maximum Weight Independent Set Problem No one knows an efficient (polynomial time) algorithm for this 1 problem Problem is NP-Complete and it is believed that there is no 2 polynomial time algorithm Brute-force algorithm: Try all subsets of vertices. Chan, Har-Peled, Hassanieh (UIUC) CS374 8 Spring 2019 8 / 22

Brute-force enumeration Algorithm to find the size of the maximum weight independent set. MaxIndSet ( G = ( V , E ) ): max = 0 for each subset S ⊆ V do check if S is an independent set if S is an independent set and w ( S ) > max then max = w ( S ) Output max Running time: suppose G has n vertices and m edges 2 n subsets of V 1 checking each subset S takes O ( m ) time 2 total time is O ( m 2 n ) 3 Chan, Har-Peled, Hassanieh (UIUC) CS374 9 Spring 2019 9 / 22

Brute-force enumeration Algorithm to find the size of the maximum weight independent set. MaxIndSet ( G = ( V , E ) ): max = 0 for each subset S ⊆ V do check if S is an independent set if S is an independent set and w ( S ) > max then max = w ( S ) Output max Running time: suppose G has n vertices and m edges 2 n subsets of V 1 checking each subset S takes O ( m ) time 2 total time is O ( m 2 n ) 3 Chan, Har-Peled, Hassanieh (UIUC) CS374 9 Spring 2019 9 / 22

A Recursive Algorithm Let V = { v 1 , v 2 , . . . , v n } . For a vertex u let N ( u ) be its neighbors. Observation v 1 : vertex in the graph. One of the following two cases is true Case 1 v 1 is in some maximum independent set. Case 2 v 1 is in no maximum independent set. We can try both cases to “reduce” the size of the problem G 1 = G − v 1 obtained by removing v 1 and incident edges from G G 2 = G − v 1 − N ( v 1 ) obtained by removing N ( v 1 ) ∪ v 1 from G MIS ( G ) = max { MIS ( G 1 ) , MIS ( G 2 ) + w ( v 1 ) } Chan, Har-Peled, Hassanieh (UIUC) CS374 10 Spring 2019 10 / 22

A Recursive Algorithm Let V = { v 1 , v 2 , . . . , v n } . For a vertex u let N ( u ) be its neighbors. Observation v 1 : vertex in the graph. One of the following two cases is true Case 1 v 1 is in some maximum independent set. Case 2 v 1 is in no maximum independent set. We can try both cases to “reduce” the size of the problem G 1 = G − v 1 obtained by removing v 1 and incident edges from G G 2 = G − v 1 − N ( v 1 ) obtained by removing N ( v 1 ) ∪ v 1 from G MIS ( G ) = max { MIS ( G 1 ) , MIS ( G 2 ) + w ( v 1 ) } Chan, Har-Peled, Hassanieh (UIUC) CS374 10 Spring 2019 10 / 22

A Recursive Algorithm Let V = { v 1 , v 2 , . . . , v n } . For a vertex u let N ( u ) be its neighbors. Observation v 1 : vertex in the graph. One of the following two cases is true Case 1 v 1 is in some maximum independent set. Case 2 v 1 is in no maximum independent set. We can try both cases to “reduce” the size of the problem G 1 = G − v 1 obtained by removing v 1 and incident edges from G G 2 = G − v 1 − N ( v 1 ) obtained by removing N ( v 1 ) ∪ v 1 from G MIS ( G ) = max { MIS ( G 1 ) , MIS ( G 2 ) + w ( v 1 ) } Chan, Har-Peled, Hassanieh (UIUC) CS374 10 Spring 2019 10 / 22

A Recursive Algorithm RecursiveMIS ( G ): if G is empty then Output 0 a = RecursiveMIS ( G − v 1 ) b = w ( v 1 ) + RecursiveMIS ( G − v 1 − N ( v n ) ) Output max( a , b ) Chan, Har-Peled, Hassanieh (UIUC) CS374 11 Spring 2019 11 / 22

Example Chan, Har-Peled, Hassanieh (UIUC) CS374 12 Spring 2019 12 / 22

Recursive Algorithms ..for Maximum Independent Set Running time: � � T ( n ) = T ( n − 1) + T n − 1 − deg ( v 1 ) + O (1 + deg ( v 1 )) where deg ( v 1 ) is the degree of v 1 . T (0) = T (1) = 1 is base case. Worst case is when deg ( v 1 ) = 0 when the recurrence becomes T ( n ) = 2 T ( n − 1) + O (1) Solution to this is T ( n ) = O (2 n ) . Chan, Har-Peled, Hassanieh (UIUC) CS374 13 Spring 2019 13 / 22

Backtrack Search via Recursion Recursive algorithm generates a tree of computation where each 1 node is a smaller problem (subproblem) Simple recursive algorithm computes/explores the whole tree 2 blindly in some order. Backtrack search is a way to explore the tree intelligently to 3 prune the search space Some subproblems may be so simple that we can stop the 1 recursive algorithm and solve it directly by some other method Memoization to avoid recomputing same problem 2 Stop the recursion at a subproblem if it is clear that there is no 3 need to explore further. Leads to a number of heuristics that are widely used in practice 4 although the worst case running time may still be exponential. Chan, Har-Peled, Hassanieh (UIUC) CS374 14 Spring 2019 14 / 22

12.1: Longest Increasing Subsequence Chan, Har-Peled, Hassanieh (UIUC) CS374 15 Spring 2019 15 / 22

Sequences Definition Sequence : an ordered list a 1 , a 2 , . . . , a n . Length of a sequence is number of elements in the list. Definition a i 1 , . . . , a i k is a subsequence of a 1 , . . . , a n if 1 ≤ i 1 < i 2 < . . . < i k ≤ n . Definition A sequence is increasing if a 1 < a 2 < . . . < a n . It is non-decreasing if a 1 ≤ a 2 ≤ . . . ≤ a n . Similarly decreasing and non-increasing . Chan, Har-Peled, Hassanieh (UIUC) CS374 16 Spring 2019 16 / 22

Sequences Example... Example Sequence: 6 , 3 , 5 , 2 , 7 , 8 , 1 , 9 1 Subsequence of above sequence: 5 , 2 , 1 2 Increasing sequence: 3 , 5 , 9 , 17 , 54 3 Decreasing sequence: 34 , 21 , 7 , 5 , 1 4 Increasing subsequence of the first sequence: 2 , 7 , 9 . 5 Chan, Har-Peled, Hassanieh (UIUC) CS374 17 Spring 2019 17 / 22

Longest Increasing Subsequence Problem Input A sequence of numbers a 1 , a 2 , . . . , a n Goal Find an increasing subsequence a i 1 , a i 2 , . . . , a i k of maximum length Example Sequence: 6, 3, 5, 2, 7, 8, 1 1 Increasing subsequences: 6, 7, 8 and 3, 5, 7, 8 and 2, 7 etc 2 Longest increasing subsequence: 3, 5, 7, 8 3 Chan, Har-Peled, Hassanieh (UIUC) CS374 18 Spring 2019 18 / 22

Longest Increasing Subsequence Problem Input A sequence of numbers a 1 , a 2 , . . . , a n Goal Find an increasing subsequence a i 1 , a i 2 , . . . , a i k of maximum length Example Sequence: 6, 3, 5, 2, 7, 8, 1 1 Increasing subsequences: 6, 7, 8 and 3, 5, 7, 8 and 2, 7 etc 2 Longest increasing subsequence: 3, 5, 7, 8 3 Chan, Har-Peled, Hassanieh (UIUC) CS374 18 Spring 2019 18 / 22

Na¨ ıve Enumeration Assume a 1 , a 2 , . . . , a n is contained in an array A algLISNaive ( A [1 .. n ] ): max = 0 for each subsequence B of A do if B is increasing and | B | > max then max = | B | Output max Running time: O ( n 2 n ) . 2 n subsequences of a sequence of length n and O ( n ) time to check if a given sequence is increasing. Chan, Har-Peled, Hassanieh (UIUC) CS374 19 Spring 2019 19 / 22