fast robustness quantification with variational bayes
play

Fast Robustness Quantification with Variational Bayes Tamara - PowerPoint PPT Presentation

Jan 22, 2024 •10 likes •1.45k views

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan Huggins, Michael I. Jordan Bayesian inference Complex, modular

  1. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) q ( θ ) KL ( q k p ( ·| x )) • VB practical success • point estimates and prediction • fast 3

  2. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  3. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  4. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  5. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  6. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  7. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  8. Variational Bayes • Variational Bayes (VB) • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast, streaming, distributed 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  9. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  10. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) • No covariance estimates 4

  11. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  12. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  13. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  14. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  15. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  16. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  17. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011]

  18. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011] 4 [Fosdick 2013; Dunson 2014; Bardenet, Doucet, Holmes 2015]

  19. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  20. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  21. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  22. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  23. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  24. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  25. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  26. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  27. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  28. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  29. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  30. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  31. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  32. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  33. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  34. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation  d �� d � Σ = dtC p ( ·| x ) ( t ) � dt T � t =0 5 [Bishop 2006] [Bishop 2006]

  35. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  36. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  37. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  38. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006] [Bishop 2006]

  39. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [Bishop 2006]

  40. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  41. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  42. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t = ( I − V H ) − 1 V 6

  43. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  44. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  45. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  46. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: 6

  47. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: E p t θ ≈ E q ∗ t θ 6

  48. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) [Bishop 2006] 6

  49. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) • LRVB estimate is exact when MFVB gives exact mean (e.g. multivariate normal) [Bishop 2006] 6

  50. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  51. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  52. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  53. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  54. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  55. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  56. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  57. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  58. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  59. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  60. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  61. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  62. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  63. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  64. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ 7

  65. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ iid σ − 2 ∼ Γ ( a, b ) k 7

  66. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ ✓ ◆ ✓✓ ◆ ◆ µ k µ iid µ iid µ 0 , Λ − 1 ∼ N ∼ N , C τ k τ τ τ 0 iid σ − 2 ∼ Γ ( a, b ) C ∼ Sep&LKJ( η , c, d ) k 7

Recommend

Fast Quantification of Uncertainty and Robustness with Variational Bayes Tamara Broderick ITT

Fast Quantification of Uncertainty and Robustness with Variational Bayes Tamara Broderick ITT

Fast Quantification of Uncertainty and Robustness with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan H. Huggins, Michael I. Jordan Bayesian inference

1.84k views • 170 slides
Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of

Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of

Another Walkthrough of Variational Bayes Bevan Jones ML for NLP Reading Group The University of Edinburgh 18 th of October, 2012 2 Variational Bayes? Bayes Bayes Theorem But the integral is intractable! Sampling Gibbs,

1.51k views • 140 slides
Variational Auto-encoders 2 VARIATIONAL AUTO-ENCODERS INTRODUCTION VARIATIONAL AUTO-ENCODERS

Variational Auto-encoders 2 VARIATIONAL AUTO-ENCODERS INTRODUCTION VARIATIONAL AUTO-ENCODERS

Lecture 3 Variational Auto-encoders 2 VARIATIONAL AUTO-ENCODERS INTRODUCTION VARIATIONAL AUTO-ENCODERS In this talk I will in some detail describe the paper of Kingma and Welling. Auto-Encoding Variational Bayes , International

706 views • 58 slides
EM & Variational Bayes Hanxiao Liu September 9, 2014 1 / 19 Outline 1. EM Algorithm 1.1

EM & Variational Bayes Hanxiao Liu September 9, 2014 1 / 19 Outline 1. EM Algorithm 1.1

EM & Variational Bayes Hanxiao Liu September 9, 2014 1 / 19 Outline 1. EM Algorithm 1.1 Introduction 1.2 Example: Mixture of vMFs 2. Variational Bayes 2.1 Introduction 2.2 Example: Bayesian Mixture of Gaussians 2 / 19 MLE by Gradient

527 views • 19 slides
Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational Inference Review Lower bound the likelihood L ( ; X ) = E q log p ( X | ) log p ( X , Z | ) q ( Z ) = E q + E q log q (

310 views • 14 slides
Applications of Variational Bayes & DAGs in Neuroimaging ECE

Applications of Variational Bayes & DAGs in Neuroimaging ECE

Applications of Variational Bayes & DAGs in Neuroimaging ECE 6504: Advanced Topics in Machine Learning Rosalyn Moran rosalynj@vtc.vt.edu

665 views • 53 slides
Improving the Robustness of Variational Optical Flow through Tensor Voting by: Hatem A.

Improving the Robustness of Variational Optical Flow through Tensor Voting by: Hatem A.

Improving the Robustness of Variational Optical Flow through Tensor Voting by: Hatem A. Rashwan, Domenec Puig, Miguel Angel Garcia presented by: Merlin Lang langmerlin@stud.uni-sb.de Milestones And Advatages in Image Analysis M athematical

297 views • 26 slides
Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational

Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational

Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational Bayes Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College

1.21k views • 116 slides
Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational

Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational

Probabilistic & Unsupervised Learning Factored Variational Approximations and Variational Bayes Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College

1.31k views • 108 slides
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations Wu Lin (UBC) June 11, 2019 Joint work with Mohammad Emtiyaz Khan (AIP, RIKEN) and Mark Schmidt (UBC) 1 / 7 Variational Inference (VI) VI

119 views • 7 slides
Switching Linear Dynamics for Variational Bayes Filtering Philip Becker-Ehmck 1 , 2 , Jan Peters 2

Switching Linear Dynamics for Variational Bayes Filtering Philip Becker-Ehmck 1 , 2 , Jan Peters 2

Switching Linear Dynamics for Variational Bayes Filtering Philip Becker-Ehmck 1 , 2 , Jan Peters 2 , Patrick van der Smagt 1 1 Machine Learning Research Lab, Volkswagen Group 2 Department of Computer Science, Technische Universit at Darmstadt

340 views • 8 slides
Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

European Geosciences Union General Assembly 2017 Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary Thermomechanical Ice Sheet (f.ETISh) model K. Bulthuis 1 , 2 , M. Arnst 1 , F. Pattyn 2 , L. Favier 2

647 views • 28 slides
Large Sample Robustness Bayes Nets with Incomplete Information Jim Smith and Ali Daneshkhah

Large Sample Robustness Bayes Nets with Incomplete Information Jim Smith and Ali Daneshkhah

Large Sample Robustness Bayes Nets with Incomplete Information Jim Smith and Ali Daneshkhah Universities of Warwick and Strathclyde Denmark PGM September 2008 Denmark PGM September 2008 1 / Jim Smith (Warwick) Robust Bayes Nets 30

395 views • 37 slides
Variational Autoencoders Tom Fletcher March 25, 2019 Talking about this paper: Diederik Kingma

Variational Autoencoders Tom Fletcher March 25, 2019 Talking about this paper: Diederik Kingma

Variational Autoencoders Tom Fletcher March 25, 2019 Talking about this paper: Diederik Kingma and Max Welling, Auto-Encoding Variational Bayes, In International Conference on Learning Representation (ICLR) , 2014. Autoencoders Input Latent

409 views • 15 slides
Fast Variational Algorithms for Statistical Network Modeling and other network modeling advances

Fast Variational Algorithms for Statistical Network Modeling and other network modeling advances

Hierarchical ERG models Fast Variational Algorithms for Statistical Network Modeling and other network modeling advances David Hunter Michael Schweinberger Duy Vu Ruth Hummel Department of Statistics, Penn State University MURI meeting, Nov

521 views • 34 slides
Level set methods for robustness measures P. V AN D OOREN CESAME, Universit e catholique de

Level set methods for robustness measures P. V AN D OOREN CESAME, Universit e catholique de

Level set methods for robustness measures P. V AN D OOREN CESAME, Universit e catholique de Louvain joint work with Y. Genin, T. Lawrence, M. Overton, J. Sreedhar, A. Tits What it is about Fast and reliable computation of robustness measures

490 views • 29 slides
Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte

Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte

Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte Helmert and Gabriele R oger University of Basel May 24, 2017 Introduction Probability Theory Inference from Full Joint Distributions Bayes

432 views • 29 slides
Variational Hamiltonian Monte Carlo via Score Matching Cheng Zhang (Joint work with Prof. Shahbaba

Variational Hamiltonian Monte Carlo via Score Matching Cheng Zhang (Joint work with Prof. Shahbaba

Background Markov chain Monte Carlo Fixed-Form Variational Bayes Variational Hamiltonian Monte Carlo Conclusion Rfrences Variational Hamiltonian Monte Carlo via Score Matching Cheng Zhang (Joint work with Prof. Shahbaba and Prof. Zhao)

451 views • 44 slides
Projected Stein variational Newton: A fast and scalable Bayesian inference method in high

Projected Stein variational Newton: A fast and scalable Bayesian inference method in high

Projected Stein variational Newton: A fast and scalable Bayesian inference method in high dimensions Peng Chen Keyi Wu, Joshua Chen, Thomas OLeary-Roseberry, Omar Ghattas Oden Institute for Computational Engineering and Sciences The University

1.16k views • 75 slides
Robustness? Robustness ? Robustness?

Robustness? Robustness ? Robustness?

Robustness? Robustness ? Robustness? Thomas Mandl

245 views • 5 slides
Where Are We? Lecture 9 Robustness through Training 1 Robustness Explicit Handling of Noise

Where Are We? Lecture 9 Robustness through Training 1 Robustness Explicit Handling of Noise

Where Are We? Lecture 9 Robustness through Training 1 Robustness Explicit Handling of Noise and Channel Variations 2 Michael Picheny, Bhuvana Ramabhadran, Stanley F. Chen Robustness via Adaptation 3 IBM T.J. Watson Research Center

256 views • 22 slides
Ludger BioQuant Chitotriose Standard for Glycan Quantitation A fast, reliable method for

Ludger BioQuant Chitotriose Standard for Glycan Quantitation A fast, reliable method for

Ludger BioQuant Chitotriose Standard for Glycan Quantitation A fast, reliable method for quantifying your glycans Highlights of the Chitotriose Standard System for Glycan Quantification Regulatory Submissions Reliable quantification

78 views • 7 slides
Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for

Nave Bayes Classifica/on Debapriyo Majumdar Data Mining Fall 2014 Indian Statistical Institute Kolkata August 14, 2014 Bayes Theorem Thomas Bayes (1701-1761) Simple form of Bayes Theorem, for two

1.04k views • 20 slides
Quantification and Quantificational Structures 7/21/17 Overview Interpreting DPs (entity

Quantification and Quantificational Structures 7/21/17 Overview Interpreting DPs (entity

Quantification and Quantificational Structures 7/21/17 Overview Interpreting DPs (entity denoting, first order quantification, generalized quantification) Potential for scope interactions Expressivity of natural language

511 views • 14 slides

More recommend