Fast Semi-differential based Submodular Function Optimization - PowerPoint PPT Presentation

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Fast Semi-differential based Submodular Function Optimization Rishabh Iyer 1 Stefanie Jegelka 2 Jeff Bilmes 1 1 University of Washington, Seattle 2 University of California, Berkeley ICML-2013 Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 1 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Outline Submodular Functions in Machine Learning 1 Convexity, Concavity & Submodular Semigradient Descent 2 Submodular Minimization 3 Submodular Maximization 4 Conclusion 5 Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 2 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Set functions f : 2 V → R { , } , , , V = , , , , V is a finite “ground” set of objects. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20

} Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Set functions f : 2 V → R { , } , , A = , A set function f : 2 V → R produces a value for any subset A ⊆ V . Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20

} Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Set functions f : 2 V → R { , } , , A = , A set function f : 2 V → R produces a value for any subset A ⊆ V . For example, f ( A ) = 22, Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20

} Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Set functions f : 2 V → R { , } , , A = , A set function f : 2 V → R produces a value for any subset A ⊆ V . For example, f ( A ) = 22, General set function optimization can be really hard! Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Gain = 1 Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Gain = 1 Gain = 0 Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Gain = 1 Gain = 0 Monotonicity: f ( A ) ≤ f ( B ) , if A ⊆ B . Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Set Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) Gain = 1 Gain = 0 Monotonicity: f ( A ) ≤ f ( B ) , if A ⊆ B . Modular function f ( X ) = � i ∈ X f ( i ) analogous to linear functions. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Maximization compute A ∗ ∈ argmax f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Maximization compute A ∗ ∈ argmax f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Sensor Placement (Krause et al, 2008) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Maximization compute A ∗ ∈ argmax f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Sensor Placement (Krause et al, 2008) Document Summarization (Lin & Bilmes, 2011) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Maximization compute A ∗ ∈ argmax f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Diversified Search (He Sensor Placement et al 2012, Kulesza & (Krause et al, 2008) Document Summarization Taskar, 2012) (Lin & Bilmes, 2011) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Minimization compute A ∗ ∈ argmin f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Minimization compute A ∗ ∈ argmin f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Image segmentation / MAP inference (Boykov & Jolly 2001, Jegelka & Bilmes 2011, Delong et al, 2012) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Minimization compute A ∗ ∈ argmin f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Clustering Image segmentation / MAP (Narasimhan & inference (Boykov & Jolly 2001, Bilmes 2011, Nagano Jegelka & Bilmes 2011, Delong et al, 2010) et al, 2012) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Submodular Function Minimization compute A ∗ ∈ argmin f ( A ) A ∈C where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently. Clustering Image segmentation / MAP Corpus Data Subset (Narasimhan & inference (Boykov & Jolly 2001, Selection (Lin & Bilmes 2011, Nagano Jegelka & Bilmes 2011, Delong Bilmes, 2011) et al, 2010) et al, 2012) Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Current State of Affairs for Submodular Optimization Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Current State of Affairs for Submodular Optimization Submodular Function Minimization Polynomial-time but too slow O ( n 5 × FuncEvalCost + n 6 ). Constrained minimization is NP-hard. Algorithms differ depending on the constraints. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20

Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Current State of Affairs for Submodular Optimization Submodular Function Submodular Function Minimization Maximization Polynomial-time but too slow NP-hard but constant-factor O ( n 5 × FuncEvalCost + n 6 ). approximable. Constrained minimization is Large class of algorithms NP-hard. – Local search, continuous greedy, bi-directional greedy, Algorithms differ depending simulated annealing etc. on the constraints. Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20