A Conditional-Gradient-Based Augmented Lagrangian Framework Alp Yurtsever alp.yurtsever@epfl.ch joint work with Olivier Fercoq & Volkan Cevher Télécom ParisTech EPFL ICML2019 - Long Beach Télécom ParisTech Ecole Polytechnique Fédérale de Lausanne ( EPFL )
<latexit sha1_base64="3gBcBo+cM9T3mVcxfXZMDy1T/mU=">ACGHicbVBNS8NAEN34WeNX1aOXxSLopSYq6FH04rGC1UJTymY7sUs3m7A7kZYQ/4UX/4oXD4p49ea/cVt70NYHA4/3ZpiZF6ZSGPS8L2dmdm5+YbG05C6vrK6tlzc2b0ySaQ51nshEN0JmQAoFdRQoZFqYHEo4TbsXQz923vQRiTqGgcptGJ2p0QkOEMrtcsHQaY61gfM+zQigYxwy5nMm8UR4g9DGPhSoK98GN9vr7XLFq3oj0Gnij0mFjFrlz+DTsKzGBRyYxp+l6KrZxpFxC4QaZgZTxHruDpqWKxWBa+eixgu5apUOjRNtSEfq74mcxcYM4tB2Ds82k95Q/M9rZhidtnKh0gxB8Z9FUSYpJnSYEu0IDRzlwBLGtbC3Ut5lmnG0Wbk2BH/y5Wlyc1j1j6qHV8eVs/NxHCWyTXbIHvHJCTkjl6RG6oSTR/JMXsmb8+S8O/Ox0/rjDOe2SJ/4Hx+A7peoMo=</latexit> <latexit sha1_base64="NSslLu+o8lefHuUQqE4Y+ElquS4=">ADFXicdVLjtMwFHXCayivDizZWLRISEhVWhagWQ2wQWIzIDpTqelUjnuTWONHZN9UVFH5CDb8ChsWIMQWiR1/g5PJjOZRrhTl6Nzr4+NjJ4UDqPobxBeuXrt+o2tm51bt+/cvdfdvr/vTGk5jLmRxk4S5kAKDWMUKGFSWGAqkXCQHL2u+wdLsE4Y/QFXBcwUy7RIBWfoqfl28DROZelyCSl2+jFawXQmwYosx/6nfqwY5pzJarKmsSsTB0gbLkmq9+tD3afCUa50Uv46H+qYBxpMxVvkEt3zgmiOSPWSjlDOYnimpeW10s9xLukP/L3i4aCUzsQRN64SYpYoVm8VOhd6eWhGqkHBixR9q3u1Fg6gpehkMW9Ajbe3Nu3/iheGlAo1cMuemw6jAWcUsCi5h3YlLBz6xI5bB1EPNFLhZ1dzqmj72zIKmxvpPI23YsysqpxbqcRP1tbdxV5NbupNS0xfzCqhixJB8+ON0lJSn179ROhCWOAoVx4wboX3SnOrL9Z/5A6PoThxSNfBvujwfDZYPRu1Nt91caxR6SR+QJGZLnZJe8IXtkTHjwOfgafA9+hF/Cb+HP8NfxaBi0ax6QcxX+/gcB9fso</latexit> (Frank & Wolfe, 1956) Conditional Gradient Method (CGM) (Hazan, 2008) (Jaggi, 2013) min x ∈ X f ( x ) . X ⊂ R n is a convex compact set . f : X → R is a smooth convex function X x k Algorithm 1 CGM for smooth minimization x k +1 Input: x 1 2 X { x : f ( x ) Æ f ( x k ) } for k = 1 , 2 , . . . , do η k = 2 / ( k + 1) ≠Ò f ( x k ) ⌦ ↵ s k = arg min x ∈ X r f ( x k ) , x x k +1 = x k + η k ( s k � x k ) end for s k
<latexit sha1_base64="NSslLu+o8lefHuUQqE4Y+ElquS4=">ADFXicdVLjtMwFHXCayivDizZWLRISEhVWhagWQ2wQWIzIDpTqelUjnuTWONHZN9UVFH5CDb8ChsWIMQWiR1/g5PJjOZRrhTl6Nzr4+NjJ4UDqPobxBeuXrt+o2tm51bt+/cvdfdvr/vTGk5jLmRxk4S5kAKDWMUKGFSWGAqkXCQHL2u+wdLsE4Y/QFXBcwUy7RIBWfoqfl28DROZelyCSl2+jFawXQmwYosx/6nfqwY5pzJarKmsSsTB0gbLkmq9+tD3afCUa50Uv46H+qYBxpMxVvkEt3zgmiOSPWSjlDOYnimpeW10s9xLukP/L3i4aCUzsQRN64SYpYoVm8VOhd6eWhGqkHBixR9q3u1Fg6gpehkMW9Ajbe3Nu3/iheGlAo1cMuemw6jAWcUsCi5h3YlLBz6xI5bB1EPNFLhZ1dzqmj72zIKmxvpPI23YsysqpxbqcRP1tbdxV5NbupNS0xfzCqhixJB8+ON0lJSn179ROhCWOAoVx4wboX3SnOrL9Z/5A6PoThxSNfBvujwfDZYPRu1Nt91caxR6SR+QJGZLnZJe8IXtkTHjwOfgafA9+hF/Cb+HP8NfxaBi0ax6QcxX+/gcB9fso</latexit> <latexit sha1_base64="3gBcBo+cM9T3mVcxfXZMDy1T/mU=">ACGHicbVBNS8NAEN34WeNX1aOXxSLopSYq6FH04rGC1UJTymY7sUs3m7A7kZYQ/4UX/4oXD4p49ea/cVt70NYHA4/3ZpiZF6ZSGPS8L2dmdm5+YbG05C6vrK6tlzc2b0ySaQ51nshEN0JmQAoFdRQoZFqYHEo4TbsXQz923vQRiTqGgcptGJ2p0QkOEMrtcsHQaY61gfM+zQigYxwy5nMm8UR4g9DGPhSoK98GN9vr7XLFq3oj0Gnij0mFjFrlz+DTsKzGBRyYxp+l6KrZxpFxC4QaZgZTxHruDpqWKxWBa+eixgu5apUOjRNtSEfq74mcxcYM4tB2Ds82k95Q/M9rZhidtnKh0gxB8Z9FUSYpJnSYEu0IDRzlwBLGtbC3Ut5lmnG0Wbk2BH/y5Wlyc1j1j6qHV8eVs/NxHCWyTXbIHvHJCTkjl6RG6oSTR/JMXsmb8+S8O/Ox0/rjDOe2SJ/4Hx+A7peoMo=</latexit> <latexit sha1_base64="83+K2bLu4OmRXK0DOuczSjX2e0=">ACoHicbVFNj9MwEHXC1K+ChwRkrUtUjlslRQhOC4shz2A6Kp0W9FU1cSZbKw6drAnQBWVv8X/4Ma/we32sB+MZOnpzbyZ8Zu0UtJRFP0Nwhs3b92+s3e3de/+g4eP2o+fnDpTW4FjYZSx0xQcKqlxTJIUTiuLUKYKJ+nyaJOfEfrpNFfaFXhvIQzLXMpgDy1aP9OclW7QmFOrUmBmneTEqgQoJrpusul48PRhwNhNPIfkgqemlpnmHGyIJAnySzqv8Zy3uom32rIEjLdXwnhT79Jo0qz3jQBULFe+s15dIUvCRbw+pVJWL32Pi+K6yoDQcbDILejlQcx7IzIWzvCSbNHuRP1oG/w6iHegw3YxXLT/JkRdYmahALnZnFU0bwB6xdSuG4ltcMKxNIPmnmoUQ3b7YGr/kLz2Q8N9Y/TXzLXlQ0UDq3KlNfuXHPXc1tyP/lZjXlb+eN1FVNqMX5oLxWnAzfXItn0qIgtfIAhDfQmycK8N6Tv2nLmxBf/fJ1cDrox6/6g5NB5/D9zo49oztsx6L2Rt2yI7ZkI2ZCJ4HR8H4FO4Hx6Hn8OT89Iw2GmesksRfv0HWJ7MHw=</latexit> (Hazan, 2008) Motivation: Solving Large-Scale SDP (Yurtsever et al., 2017) min x ∈ X f ( x ) . X ⊂ R n is a convex compact set . f : X → R is a smooth convex function When X is PSD-cone with bounded trace → lmo is cheap (Arithmetic Scalability) → updates are rank-1 (Storage Scalability)
<latexit sha1_base64="83+K2bLu4OmRXK0DOuczSjX2e0=">ACoHicbVFNj9MwEHXC1K+ChwRkrUtUjlslRQhOC4shz2A6Kp0W9FU1cSZbKw6drAnQBWVv8X/4Ma/we32sB+MZOnpzbyZ8Zu0UtJRFP0Nwhs3b92+s3e3de/+g4eP2o+fnDpTW4FjYZSx0xQcKqlxTJIUTiuLUKYKJ+nyaJOfEfrpNFfaFXhvIQzLXMpgDy1aP9OclW7QmFOrUmBmneTEqgQoJrpusul48PRhwNhNPIfkgqemlpnmHGyIJAnySzqv8Zy3uom32rIEjLdXwnhT79Jo0qz3jQBULFe+s15dIUvCRbw+pVJWL32Pi+K6yoDQcbDILejlQcx7IzIWzvCSbNHuRP1oG/w6iHegw3YxXLT/JkRdYmahALnZnFU0bwB6xdSuG4ltcMKxNIPmnmoUQ3b7YGr/kLz2Q8N9Y/TXzLXlQ0UDq3KlNfuXHPXc1tyP/lZjXlb+eN1FVNqMX5oLxWnAzfXItn0qIgtfIAhDfQmycK8N6Tv2nLmxBf/fJ1cDrox6/6g5NB5/D9zo49oztsx6L2Rt2yI7ZkI2ZCJ4HR8H4FO4Hx6Hn8OT89Iw2GmesksRfv0HWJ7MHw=</latexit> <latexit sha1_base64="NSslLu+o8lefHuUQqE4Y+ElquS4=">ADFXicdVLjtMwFHXCayivDizZWLRISEhVWhagWQ2wQWIzIDpTqelUjnuTWONHZN9UVFH5CDb8ChsWIMQWiR1/g5PJjOZRrhTl6Nzr4+NjJ4UDqPobxBeuXrt+o2tm51bt+/cvdfdvr/vTGk5jLmRxk4S5kAKDWMUKGFSWGAqkXCQHL2u+wdLsE4Y/QFXBcwUy7RIBWfoqfl28DROZelyCSl2+jFawXQmwYosx/6nfqwY5pzJarKmsSsTB0gbLkmq9+tD3afCUa50Uv46H+qYBxpMxVvkEt3zgmiOSPWSjlDOYnimpeW10s9xLukP/L3i4aCUzsQRN64SYpYoVm8VOhd6eWhGqkHBixR9q3u1Fg6gpehkMW9Ajbe3Nu3/iheGlAo1cMuemw6jAWcUsCi5h3YlLBz6xI5bB1EPNFLhZ1dzqmj72zIKmxvpPI23YsysqpxbqcRP1tbdxV5NbupNS0xfzCqhixJB8+ON0lJSn179ROhCWOAoVx4wboX3SnOrL9Z/5A6PoThxSNfBvujwfDZYPRu1Nt91caxR6SR+QJGZLnZJe8IXtkTHjwOfgafA9+hF/Cb+HP8NfxaBi0ax6QcxX+/gcB9fso</latexit> <latexit sha1_base64="NSslLu+o8lefHuUQqE4Y+ElquS4=">ADFXicdVLjtMwFHXCayivDizZWLRISEhVWhagWQ2wQWIzIDpTqelUjnuTWONHZN9UVFH5CDb8ChsWIMQWiR1/g5PJjOZRrhTl6Nzr4+NjJ4UDqPobxBeuXrt+o2tm51bt+/cvdfdvr/vTGk5jLmRxk4S5kAKDWMUKGFSWGAqkXCQHL2u+wdLsE4Y/QFXBcwUy7RIBWfoqfl28DROZelyCSl2+jFawXQmwYosx/6nfqwY5pzJarKmsSsTB0gbLkmq9+tD3afCUa50Uv46H+qYBxpMxVvkEt3zgmiOSPWSjlDOYnimpeW10s9xLukP/L3i4aCUzsQRN64SYpYoVm8VOhd6eWhGqkHBixR9q3u1Fg6gpehkMW9Ajbe3Nu3/iheGlAo1cMuemw6jAWcUsCi5h3YlLBz6xI5bB1EPNFLhZ1dzqmj72zIKmxvpPI23YsysqpxbqcRP1tbdxV5NbupNS0xfzCqhixJB8+ON0lJSn179ROhCWOAoVx4wboX3SnOrL9Z/5A6PoThxSNfBvujwfDZYPRu1Nt91caxR6SR+QJGZLnZJe8IXtkTHjwOfgafA9+hF/Cb+HP8NfxaBi0ax6QcxX+/gcB9fso</latexit> <latexit sha1_base64="3gBcBo+cM9T3mVcxfXZMDy1T/mU=">ACGHicbVBNS8NAEN34WeNX1aOXxSLopSYq6FH04rGC1UJTymY7sUs3m7A7kZYQ/4UX/4oXD4p49ea/cVt70NYHA4/3ZpiZF6ZSGPS8L2dmdm5+YbG05C6vrK6tlzc2b0ySaQ51nshEN0JmQAoFdRQoZFqYHEo4TbsXQz923vQRiTqGgcptGJ2p0QkOEMrtcsHQaY61gfM+zQigYxwy5nMm8UR4g9DGPhSoK98GN9vr7XLFq3oj0Gnij0mFjFrlz+DTsKzGBRyYxp+l6KrZxpFxC4QaZgZTxHruDpqWKxWBa+eixgu5apUOjRNtSEfq74mcxcYM4tB2Ds82k95Q/M9rZhidtnKh0gxB8Z9FUSYpJnSYEu0IDRzlwBLGtbC3Ut5lmnG0Wbk2BH/y5Wlyc1j1j6qHV8eVs/NxHCWyTXbIHvHJCTkjl6RG6oSTR/JMXsmb8+S8O/Ox0/rjDOe2SJ/4Hx+A7peoMo=</latexit> <latexit sha1_base64="191mwyDnT7yA3kobj1EO9egnCxM=">ACPnicbVA9j9NAEF3fF8H3gTlKmtVFJ+Uay84hQRmgQaIJ0uUuUhxF6804W9tnbHKJHl/DEafgMdJQ0FCNFSsk5S3CU30mqf3punmXlxLoXBIPju7O0fHB49aTx1j09Oz5z89vTVZoDj2eyUz3Y2ZACgU9FCihn2tgaSzhLp69r/W7z6CNyNQNLnIYpmyiRCI4Q0uNvF5UqLHVAcs5jYSiUcpwypks+1VRghzLFOhqspdLt2kNb+q/zVtfPRX9Nst58dq5DUDP1gV3QXhBjTJproj71s0zniRgkIumTGDMhxWDKNgkuo3KgwkDM+YxMYWKhYCmZYrs6v6KVlxjTJtH0K6Yq97yhZaswijW1nvaLZ1mryMW1QYPJmWAqVFwiKrwclhaSY0TpLOhYaOMqFBYxrYXelfMo042gTdW0I4fbJu+C27YfXfvTq2bn3SaOBnlJLkiLhOQ16ZAPpEt6hJMv5Af5RX47X52fzh/n7p1z9l4XpAH5fz7D6Dzr+U=</latexit> (Hazan, 2008) Motivation: Solving Large-Scale SDP (Yurtsever et al., 2017) x ∈ X f ( x ) min . X ⊂ R n is a convex compact set . f : X → R is a smooth convex function When X is PSD-cone with bounded trace → lmo is cheap (Arithmetic Scalability) → updates are rank-1 (Storage Scalability) x ∈ X f ( x ) s.t. Ax ∈ K min This paper: X → R A new CGM-type method . A : X → R d is a given linear map based on augmented Lagrangian . K is a simple convex set
Recommend
More recommend
