Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects ≥ + + Concave Aspects of Submodular Functions International Symposium on Information Theory 2020 Rishabh Iyer 1 and Jeffrey A. Bilmes 2 1 Department of CS, University of Texas at Dallas 2 Department of ECE, University of Washington 21st - 26th June, 2020 R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 1 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Outline Introduction 1 Two sides of Submodularity 2 Submodular Polyhedra 3 Submodular Sub-differentials and Super-differentials 4 Other Convex/Concave Aspects 5 R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 2 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Outline Introduction 1 Two sides of Submodularity 2 Submodular Polyhedra 3 Submodular Sub-differentials and Super-differentials 4 Other Convex/Concave Aspects 5 R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 3 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodular Functions Special class of set functions. f ( A ∪ v ) − f ( A ) ≥ f ( B ∪ v ) − f ( B ) , if A ⊆ B (1) f = # of distinct colors of balls in the urn. Gain = 1 Gain = 0 R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 4 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Alternate definition – Submodular Functions A function f : 2 V → R is submodular if: + ≥ + f ( A ) f ( B ) f ( A ∪ B ) f ( A ∩ B ) = f ( A r ) + 2 f ( C ) + f ( B r ) = f ( A r ) + f ( C ) + f ( B r ) = f ( A ∩ B ) ≥ + + R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 5 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Alternate definition – Submodular Functions A function f : 2 V → R is submodular if: + ≥ + f ( A ) f ( B ) f ( A ∪ B ) f ( A ∩ B ) = f ( A r ) + 2 f ( C ) + f ( B r ) = f ( A r ) + f ( C ) + f ( B r ) = f ( A ∩ B ) ≥ + + Submodularity has been widely used: non-additive measure theory, economics, game theory, statistical physics and thermodynamics, electrical networks, operations research, information theory and machine learning . R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 5 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity and Information Theory Given a set of random variables, X 1 , · · · , X n , the Entropy of a Set of random variables: H ( X A ) is a submodular function. R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity and Information Theory Given a set of random variables, X 1 , · · · , X n , the Entropy of a Set of random variables: H ( X A ) is a submodular function. Similarly, the Mutual Information between a set of variables and its complement: I ( X A ; X V \ A ) is symmetric submodular. R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity and Information Theory Given a set of random variables, X 1 , · · · , X n , the Entropy of a Set of random variables: H ( X A ) is a submodular function. Similarly, the Mutual Information between a set of variables and its complement: I ( X A ; X V \ A ) is symmetric submodular. Mutual Information between a set of features X A and a class C : I ( X A ; C ) is submodular for certain models (e.g. Naive Bayes) R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity and Information Theory Given a set of random variables, X 1 , · · · , X n , the Entropy of a Set of random variables: H ( X A ) is a submodular function. Similarly, the Mutual Information between a set of variables and its complement: I ( X A ; X V \ A ) is symmetric submodular. Mutual Information between a set of features X A and a class C : I ( X A ; C ) is submodular for certain models (e.g. Naive Bayes) Many Mutual Information based Feature Selection algorithms are submodular! R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity and Information Theory Given a set of random variables, X 1 , · · · , X n , the Entropy of a Set of random variables: H ( X A ) is a submodular function. Similarly, the Mutual Information between a set of variables and its complement: I ( X A ; X V \ A ) is symmetric submodular. Mutual Information between a set of features X A and a class C : I ( X A ; C ) is submodular for certain models (e.g. Naive Bayes) Many Mutual Information based Feature Selection algorithms are submodular! Submodular Functions strictly generalize Entropy as Combinatorial Information Functions and are becoming increasingly applicable in machine learning applications. R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Facets of Submodularity Coverage ¡ F ( A ) = ∪ s ∈ A area( s ) Diversity ¡ F ( A ) = log det ( L A ) Informa2on ¡ Y F ( A ) = H ( X A ) X 1 X 2 X 3 X 4 ¡ R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 7 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Outline Introduction 1 Two sides of Submodularity 2 Submodular Polyhedra 3 Submodular Sub-differentials and Super-differentials 4 Other Convex/Concave Aspects 5 R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 8 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Convex Functions and Tight Subgradients f(x) f b (x) f b (b) = f(b) f b (a) ≤ f(a) x b A convex function f has a subgradient at any in-domain point b , namely there exists f b such that f ( x ) − f ( b ) ≥ � f b , x − b � , ∀ x . (2) we have f b ( x ) = f ( b ) + � f b , x − b � We have that f ( x ) is convex, f b ( x ) is affine, and can be a tight subgradient (tight at b , affine lower bound on f ( x )) for all b R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 9 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Concave Functions and Tight Supergradients f b (x) f b (b) = f(b) f b (a) ≥ f(a) f(x) x b A concave f has a supergradient at any in-domain point b , namely there exists f b such that f ( x ) − f ( b ) ≤ � f b , x − b � , ∀ x . (3) we have f b ( x ) = f ( b ) + � f b , x − b � We have that f ( x ) is concave, f b ( x ) is affine, and can be a tight supergradient (tight at b , affine upper bound on f ( x )) for all b R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 10 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Two Sides of Submodularity R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 11 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Two sides of Submodularity Convex aspects (Fujishige (1984, Concave aspects (Vondrak (2007), 2005), Frank (1982)) This Work) Minimization: Poly-time. Max: constant-factor approx! Convex continuous extension - Multilinear extension - concave Lov´ asz extension. in a direction. Subgradients and Supergradients and Subdifferential. Superdifferential. Convex duality, discrete Under restricted settings, seperation etc. duality, separation etc. Optimality Conditions for Optimality Conditions for Submodular Min Submodular Max R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 12 / 31
Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects Submodularity, Convexity and Concavity Many interesting polyhedral and algorithmic connections between convexity and submodularity well known over the years. R. Iyer & J. Bilmes ISIT 2020: Concave Aspects of Submodular Functions page 13 / 31
Recommend
More recommend
