� 1 FAST UNCERTAINTY ESTIMATES AND BAYESIAN MODEL AVERAGING OF DNNS WESLEY MADDOX JOINT WORK WITH TIMUR GARIPOV , PAVEL IZMAILOV , DMITRY VETROV , ANDREW GORDON WILSON
� 2 SUMMARY ‣ Stochastic Weight Averaging (Izmailov et al, UAI, 2018) computes first moment of weights given from SGD iterates with a modified learning rate schedule. ‣ We propose to keep the variance as well to form a Gaussian approximation in weight space. ‣ Sample from Gaussian to compute Bayesian model averages and estimate uncertainty. ‣ Theoretically motivated from results on SGD & relation of iterates to Gaussian distribution (Ruppert, 1992 and Mandt et al, 2017).
<latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> � 3 APPROXIMATE BAYESIAN INFERENCE ‣ Why? ‣ Compute intractable integrals p ( y ∗ | y ) = E p ( θ | y ) ( p ( y | θ )) ‣ Uncertainty quantification ‣ How? ‣ Laplace: p ( θ | y ) ≈ N ( θ MAP , ( H ( θ MAP ) + λ I ) − 1 ) ‣ Variational Bayes: p ( θ | y ) ≈ N ( µ, S ) ‣ Markov Chain Monte Carlo
Recommend
More recommend