From Local SGD to Local Fixed-Point Methods for Federated Learning - PowerPoint PPT Presentation

KAUST From Local SGD to Local Fixed-Point Methods for Federated Learning Laurent Condat King Abdullah University of Science and Technology (KAUST), 
 Thuwal, Saudi Arabia Grigory 
 Dmitry 
 Elnur 
 Peter 
 Malinovsky Kovalev Gasanov Richtárik Fixed-Point Methods with Local Steps 1 / 22

KAUST Distributed Algorithms Master Node 1 Node 2 ... Node M Fixed-Point Methods with Local Steps 2 / 22

<latexit sha1_base64="wsZp0NAEfBi+CuX+vq05sOd25fY=">AFr3icpVTbtNAEJ0WDCVcmtJHXiLSAhVhJQSx+Q2sIDL4giNU2lJgTb2SRWfJMvhWDlW3iFT+IP4C84M9mUtCEQxFq2Z8/MOTsza68deW6SVqvfVlavXTdu3Fy7Vbh95+69eLG/ZMkzGJHNZzQC+NT20qU5waqkbqp06jWFm+7amPXzB/ua5ihM3DI7TUaTavtUP3J7rWCmgTnEzjMzCVsu30oFjefnxuFPf6hTLVbMqozRv1LRJj2Owo3VQ2pRl0JyKCOfFAWUwvbIogTXGdWoShGwNuXAYliu+BWNqQBuhiFCAvoEM8+ZmcaDTBnzUTYDlbxcMdglqgCToi4GDavVhJ/JsqMLtLORZNzG+Ftay0faEoDoH/jTSOX5XEtKfXomdTgoqZIEK7O0SqZdIUzL81UlUIhAsZ2F/4YtiPMaZ9Lwkmkdu6tJf7vEskozx0dm9GP2RZQZ6JqLBil5j10zaQ84FZD0A5tE5nqnetVhUFX2QrvShwC8HL6edIrGQFRMud9YW6OdQdQYpXKwk4nojb68Xdnc2tLdzeFd4kB1fy/fUN5cC5Z0OpSu5xdC0xeLqd2kb76X07GBZPJdX1aqrDickp9yYk7/Dstk3ZmsnopGp4wFb3Wlb/RXw7PWPux7IwlWr6sPNb/Yx/49gX2r4oHVxQP/ltxwnPl/Ul3syV/akgfYc/7sYacW3s8di5OqXnjpG7WnphP39bL+4f6BFujB/SQHuGU2qV9ekVH1ECmI/pMX+irUTOaxjvj/SR0dUVzNunSMNyfxTkgyA=</latexit> <latexit sha1_base64="r8K4prBOVR9ZWwMNyJc5HoJ+lzs=">AFr3icpVTbtNAEJ0GDCVcmtJHXiLSAhVlJQSx+Q2sIDL4giNU2lJgTb2SRWfJMvhWDlW3iFT+IP4C84M9mUtCEQxFq2Z8/MOTsza68deW6S1mrfVgrXrhs3bq7eKt6+c/feWmn9/kSZrGjGk7ohfGpbSXKcwPVSN3U6dRrCzf9lTHr5gf/NcxYkbBsfpKFJt3+oHbs91rBRQp7QRmZxs+Vb6cCxvPx43KlvdkqVmlmTUZ436tqokB5H4XrhkFrUpZAcysgnRQGlsD2yKMF1RnWqUQSsTmwGJYrfkVjKoKbIUohwgI6xLOP2ZlGA8xZMxG2g1U83DGYZaqCEyIuhs2rlcWfiTKji7Rz0eTcRnjbWsHmtIA6N9408hleVxLSj16JjW4qCkShKtztEomXeHMyzNVpVCIgLHdhT+G7Qhz2ueycBKpnXtrif+7RDLKc0fHZvTjD1lWkWciKqzYpefYNZP2kHMRWQ+AeXSOZ6p3LRZVR+k670IQAvh68neJ6RkCUzHlfmJtj3QGUWKW6sNOJqM32enF3Z3NrC7d3hTfJwZV8f31DOXDu2VBq6UpuMTRtsbj6XdrCm+/ldGwgmXzXl5WmKqy4nFJfcuIO/07LpJ2ZrF6KhidMRa915W/0l8Mz1n4sO2OJli8rj/X/2Ae+dYH9q+LBFcWD/1ac8Fx5f9LdbMmfGtJH2PN+rCHn1h6PnYtTat42TbrT8ynb7cr+4f6BFulB/SQHuGU2qV9ekVH1ECmI/pMX+irUTeaxjvj/S0sKI5G3RpGO5PwAEgxw=</latexit> <latexit sha1_base64="neAL1APoDnsc0+WSH4Mo0NxUsm4=">AFr3icpVTbtNAEJ0GDCXcUvrIS0QaCaHKSgpq6QNSW3jgpaJITVOpCcF2NokV3+RLIVj5Fl7hk/gD+AvOTDYlbQgEsZbt2TNzs7M2mtHnpuktdq3lcK168aNm6u3irfv3L13v7T24CQJs9hRDSf0wvjUthLluYFqpG7qdMoVpZve6pD1+yv3mu4sQNg+N0FKm2b/UDt+c6VgqoU1oPI7O40fKtdOBYXn487hxudEqVmlmTUZ436tqokB5H4VrhgFrUpZAcysgnRQGlsD2yKMF1RnWqUQSsTmwGJYrfkVjKoKbIUohwgI6xLOP2ZlGA8xZMxG2g1U83DGYZaqCEyIuhs2rlcWfiTKji7Rz0eTcRnjbWsHmtIA6N9408hleVxLSj16LjW4qCkShKtztEomXeHMyzNVpVCIgLHdhT+G7Qhz2ueycBKpnXtrif+7RDLKc0fHZvTjD1lWkWciKqzYpRfYNZN2kXMRWQ+AeXSOZ6p3LRZVR+k670IQAvh68neJ6RkCUzHlfmJtj3QGUWKW6sNOJqM32enF3Z3NrC7d3hTfJwZV8f31DOXDu2VBq6UpuMTRtsbj6HdrEm+/ldGwgmXzXl5WmKqy4nFJfcuIO/07LpO2ZrF6JhidMRYe68jf6y+EZaz+RnbFEy5eVx/p/7APfvMD+VXH/iuL+fytOeK68P+lutuRPDekj7Hk/1pBza5fH9sUpNW+cbJn1p+azt1uVvQN9gq3SQ3pEj3FK7dAevaYjaiDTEX2mL/TVqBtN453xfhJaWNGcdbo0DPcnUjAg4w=</latexit> KAUST Distributed Algorithms Master op. T 1 op. T 2 op. T M Fixed-Point Methods with Local Steps 3 / 22

<latexit sha1_base64="xlMn1nKprFMTIJxZUGkB+Sv94+A=">AFJ3iclVPbtNAEB2XACVcmoJ4QgKLKFKpqigpiFKkShXwEtFQU1bqWmjtb1JVvENX0qC5Q/gU/gOPoA3BI+8wRO/wNmJ2waicNko9syZOWdnZr1W6Ko4aTS+GHPnSucvXJy/VL585eq1hcri9d04SCNbtuzADaJ9S8TSVb5sJSpx5X4YSeFZrtyzBk91fO9YRrEK/J1kFMpDT/R81VW2SAB1Ku/Kptm2ZE/5mXydMrica8wVlnSzHbY9kfRtob3Hw7by2bes7FV+5MAO4yQw291I2Fkz7Zysx2nXidTG838iN0zOtB8aXivrkWl70xuWe5Uqo16g5c5bTQLo0rF2g4WjQ/UJocCsikljyT5lMB2SVCM3wE1qUEhsEPKgEWwFMcl5VQGN0WRIYAOsCzBy+DHSAnQH5OJtUQD5AZwdbKJsdTVtHobB2BmnQdI7ytQsDmlAf6N94J5n/yjsAmlCXHnG3CrMIGdFTsQuVlCegKzcnukqgEALTtoN4BNtm5slMTebE3Lueo+D4N87UqPbtIjel73+osoY6Y1bRig5t4ITqtI6ay6i6D8ylYzyT4oQiVpX0hqfu8Rx8DLEujwp3c8IiGRfn4vmZti3DyWtUps56ZjVJmc9e7qTtR0yt/sb1yD4nrPvqEMuJ7ZgHtxuLYImhZbuvs1WsFb/8c6z5jvMkvSVrHi+KMtKd1l3kGgrU83jUvIe8JVT7H8VxzF7d8U8aohZwh7Ol4zjd3Xa+Hp/d02thdrTfv1x+8XK1uPinu8Dzdoru0hHu6Rpv0nLaphTp/GDeN28ad0vSx9Kn0udx6pxRcG7QL6v09SfUvR6L</latexit> KAUST Distributed fixed-point problem We define the average operator M T : x 2 R d 7! 1 X T i ( x ). M i =1 Fixed-Point Methods with Local Steps 4 / 22

<latexit sha1_base64="XF2rZ5bA65PhacZ6aZ9h7KOR5k=">AE1HiclVPLbtNAFL0pBkp4pbBkY5FGQqiKkoJaukCqgAUb1IJIWqlJq7E9iUfxS/a4NJisEFu+gK+BJfwAfwB/wZkbtwSi8Jgonjvn3nN854zHSQKV6Vbra2XpnHX+wsXlS9XLV65eu15budHN4jx1ZceNgzjd0QmAxXJjlY6kPtJKkXoBHLPGT02+b1jmWYqjl7qcSL7oRhGaqBcoQEd1TZ28tQexiKwVWbr2B6oyLNXTw57mRZpT0W9UGjfcYoXk0Nv1c5y17e1L/Rrd5qtnjY80G7DOpUjt14pfKJeuRTC7lFJKkiDTigARl+B1Qm1qUAOtTASxFpDgvaUJVcHNUSVQIoCM8h1gViGPUxKifkE0N5GNUpoiNs35nFUMulhHoCfTxizU2qFQDX5QP/GO638V94BUE0DesC7VfAiYcS4pYqOTtgOrdndqWhkAzsYd8ithl5qmnNnMy3rvxUXD+G1ca1Kzdsjan73/osoE+M1Yxih49xAk1aQs9V9G1DygYzx1eUIpq0p6xa6H7EMEXoHcgJ0y+xkDkbw252K4Bd7rQ8moNBY6nbHarNeL3Z3trc/cwW+8aQ+K+/35DRXAjWcj3ovHvaXQdDgyu9+kNczmP9V5wvyAWZKele/YKc/IrIzuXfZAsFbIb52UX/kQ+NoZ9r+KU57i+TXflCnqoOYE8Xx+wjd3y4yNs3s6H3TXm+17zfvP1+vbj8o7vEy36DbdwT3dpG16SrvUQZ8f6CN9pi9W13pjvbXeTUuXKiXnJv0yrPc/APS/AG4=</latexit> <latexit sha1_base64="aMiFpDzQdxv8lXmwkSntVdGhmrw=">AE6niclVPLbtNAFL0pBkp4NIUlG4soUqmqKCmI0kWlCliwAFGkpq3UhGpsTxIrfmGPS4OVn2CH2PIF/AI/AUtYwV9w5npaAlF4TGL7zrn3HN8543GSwM9Uq/W5snDOn/h4uKl6uUrV68t1Zav72Vxnrqy48ZBnB4IpOBH8mO8lUgD5JUitAJ5L4zeqjz+8cyzfw42lXjRPZCMYj8vu8KBeio9qRatW38u4c+FEhX+acWJ1oLBRq6Iqg2J2snLzoZkqkt+0t24RNXSEjb4pzVKu3mi0e9mzQNkGdzNiJlysfqUsexeRSTiFJikghDkhQht8htalFCbAeFcBSRD7nJU2oCm6OKokKAXSE+wCzAnGMmhj1E7KpgXyMyhSxVrY5n7OKRufrCPSk+xj6RitEKiIdC/8U4r/5V3CFRn+7zan14kTCiXGNSs4O6M7tqVUpKCTAdOwhnyJ2mXnqc2cjNeufRSc/8aVGtVz19Tm9P0PXTbQZ8YqWtGjLexQkzbRcxVdD4EFdIy7MjuUsqkV+x6yD5E4BXI9dkpvZ4xEMlzvS+aW+C9QyhplcZcpzNWm/Z6vrvTvfWY2/+NV/bgc78/v6ECuPZsxGvxuLcUmg5HevUbtIanvkqdR8wPmCXpqXnHM7NHeqZ1V9kDwVohv3VivIB8LUz7H8VS57Pz9d8UkrUQc0J4tl8eXI39bh3dk5ng731ZvtO8+7z9fr2A3OGF+km3aIVnNMN2qbHtEMd9PmBPtEX+moF1hvrfWuLF2oGM4N+mVY738AgsUHA=</latexit> <latexit sha1_base64="xlMn1nKprFMTIJxZUGkB+Sv94+A=">AFJ3iclVPbtNAEB2XACVcmoJ4QgKLKFKpqigpiFKkShXwEtFQU1bqWmjtb1JVvENX0qC5Q/gU/gOPoA3BI+8wRO/wNmJ2waicNko9syZOWdnZr1W6Ko4aTS+GHPnSucvXJy/VL585eq1hcri9d04SCNbtuzADaJ9S8TSVb5sJSpx5X4YSeFZrtyzBk91fO9YRrEK/J1kFMpDT/R81VW2SAB1Ku/Kptm2ZE/5mXydMrica8wVlnSzHbY9kfRtob3Hw7by2bes7FV+5MAO4yQw291I2Fkz7Zysx2nXidTG838iN0zOtB8aXivrkWl70xuWe5Uqo16g5c5bTQLo0rF2g4WjQ/UJocCsikljyT5lMB2SVCM3wE1qUEhsEPKgEWwFMcl5VQGN0WRIYAOsCzBy+DHSAnQH5OJtUQD5AZwdbKJsdTVtHobB2BmnQdI7ytQsDmlAf6N94J5n/yjsAmlCXHnG3CrMIGdFTsQuVlCegKzcnukqgEALTtoN4BNtm5slMTebE3Lueo+D4N87UqPbtIjel73+osoY6Y1bRig5t4ITqtI6ay6i6D8ylYzyT4oQiVpX0hqfu8Rx8DLEujwp3c8IiGRfn4vmZti3DyWtUps56ZjVJmc9e7qTtR0yt/sb1yD4nrPvqEMuJ7ZgHtxuLYImhZbuvs1WsFb/8c6z5jvMkvSVrHi+KMtKd1l3kGgrU83jUvIe8JVT7H8VxzF7d8U8aohZwh7Ol4zjd3Xa+Hp/d02thdrTfv1x+8XK1uPinu8Dzdoru0hHu6Rpv0nLaphTp/GDeN28ad0vSx9Kn0udx6pxRcG7QL6v09SfUvR6L</latexit> KAUST Distributed fixed-point problem We define the average operator M T : x 2 R d 7! 1 X T i ( x ). M i =1 Our goal is to find x ? ∈ R d such that T ( x ? ) = x ? . Fixed-Point Methods with Local Steps 4 / 22