fast quantification of uncertainty and robustness with
play

Fast Quantification of Uncertainty and Robustness with Variational - PowerPoint PPT Presentation

Nov 30, 2023 •131 likes •1.84k views

Fast Quantification of Uncertainty and Robustness with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan H. Huggins, Michael I. Jordan Bayesian inference

  1. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) q ( θ ) KL ( q k p ( ·| x )) • VB practical success • point estimates and prediction • fast 3

  2. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  3. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  4. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  5. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  6. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  7. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  8. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast, streaming, distributed 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  9. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  10. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) • No covariance estimates 4

  11. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  12. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  13. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  14. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  15. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  16. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  17. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011]

  18. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011] 4 [Fosdick 2013; Dunson 2014; Bardenet, Doucet, Holmes 2015]

  19. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  20. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  21. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  22. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  23. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  24. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  25. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  26. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  27. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  28. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  29. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  30. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  31. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  32. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  33. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  34. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  35. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  36. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation  d �� d � Σ = dtC p ( ·| x ) ( t ) � [Bishop 2006] dt T � t =0 5 [see also Opper, Winther 2003]

  37. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  38. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  39. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  40. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  41. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  42. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  43. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  44. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t = ( I − V H ) − 1 V 6

  45. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  46. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  47. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  48. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: 6

  49. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: E p t θ ≈ E q ∗ t θ 6

  50. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) [Bishop 2006] 6

  51. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) • LRVB estimate is exact when MFVB gives exact mean (e.g. multivariate normal) [Bishop 2006] 6

  52. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  53. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  54. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  55. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  56. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  57. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  58. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  59. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  60. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  61. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  62. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  63. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  64. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  65. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  66. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ 7

  67. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ iid σ − 2 ∼ Γ ( a, b ) k 7

  68. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ ✓ ◆ ✓✓ ◆ ◆ µ k µ iid µ iid µ 0 , Λ − 1 ∼ N ∼ N , C τ k τ τ τ 0 iid σ − 2 ∼ Γ ( a, b ) C ∼ Sep&LKJ( η , c, d ) k 7

Recommend

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan Huggins, Michael I. Jordan Bayesian inference Complex, modular

1.45k views • 144 slides
Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

European Geosciences Union General Assembly 2017 Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary Thermomechanical Ice Sheet (f.ETISh) model K. Bulthuis 1 , 2 , M. Arnst 1 , F. Pattyn 2 , L. Favier 2

647 views • 28 slides
Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1

Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1

Motivation Uncertainty Quantification Case Study Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1 1 Computational Science Lab Universities of Amsterdam Quantification of Uncertainty: Improving

268 views • 11 slides
Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred

Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred

Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred Optimization and Uncertainty Quantification Dept. Sandia National Laboratories, Albuquerque, NM NIST UQ Workshop Boulder, CO; August 1-4, 2011

512 views • 33 slides
Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair

Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair

Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair of Risk, Safety and Uncertainty Quantification ETH Z urich CEMRACS Summer School on Numerical methods for stochastic models: control,

1.01k views • 82 slides
The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are

The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are

Army Conference on Applied Statistics Santa Fe, NM October 25, 2001 The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are We Adding More Uncertainty or More Understanding?) or More Understanding?)

374 views • 21 slides
Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L.

Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L.

Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L. Muhanna Center for Reliable Engineering Computing (REC) Georgia Institute of Technology ifip -Working Conference on Uncertainty Quantification in

1.57k views • 52 slides
A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari

A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari

A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari Department of Aerospace Engineering, University of Bristol, Bristol, U.K. Email: S.Adhikari@bristol.ac.uk Uncertainty Quantification p.1/43 Newport, RI,

547 views • 43 slides
46.1 Introduction chapter overview: 46. Introduction and Quantification 47. Representation

46.1 Introduction chapter overview: 46. Introduction and Quantification 47. Representation

Foundations of Artificial Intelligence May 24, 2017 46. Uncertainty: Introduction and Quantification Foundations of Artificial Intelligence 46.1 Introduction 46. Uncertainty: Introduction and Quantification 46.2 Probability Theory Malte

213 views • 8 slides
Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our Contribution This

Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our Contribution This

Hybrid Multilevel Monte Carlo Simulation of Stochastic Reaction Networks Alvaro Moraes joint work with Ra ul Tempone and Pedro Vilanova http://sri-uq.kaust.edu.sa Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our

699 views • 39 slides
Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian,

Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian,

W ESTFLISCHE W ILHELMS -U NIVERSITT M NSTER Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian, C. Engwer, J. Fahlke, M. Geveler, D. living knowledge Gddeke, O. Iliev, O. Ippisch, R. Milk, J.

327 views • 32 slides
Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for

Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for

Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for LULUCF Marcel van Oijen & Amanda Thomson Centre for Ecology and Hydrology Edinburgh, U.K. Contents 1. Current methodology used to make the

299 views • 25 slides
THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of

THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of

THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of Massachusetts Amherst Quantissima III, Venice, August 2019 Work supported by NSF and AFOSR 1 Collaborators on related projects Paul Dupuis (Brown

827 views • 34 slides
Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and

Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and

Introduction Related work Problem Description and Approach Results and Conclusion Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and Nonparametric Noise Models Tushar Athawale Department of Computer

1.32k views • 115 slides
Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory

Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory

Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory Miroslav Stoyanov Oak Ridge National Laboratory Clayton G. Webster Oak Ridge National Laboratory Quantification of Uncertainty: Improving Efficiency

679 views • 43 slides
Bayesian calibration and uncertainty quantification of computer models Application to the

Bayesian calibration and uncertainty quantification of computer models Application to the

Bayesian calibration and uncertainty quantification of computer models Application to the simulation of astrophysical radiative shocks Jean Giorla 1 , J. Garnier 2 , . Falize 1 , B. Loupias 1 1 CEA/DAM/DIF, F-91297, Arpajon, France 2

780 views • 30 slides
Uncertainty quantification EMA workshop on qualification and reporting of physiologically- based

Uncertainty quantification EMA workshop on qualification and reporting of physiologically- based

Uncertainty quantification EMA workshop on qualification and reporting of physiologically- based pharmacokinetic (PBPK) modelling and simulation Session III EMA 2016-11-21 Ine Skottheim Rusten Disclaimer: The views expressed in this

148 views • 10 slides
Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte

Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte

Foundations of Artificial Intelligence 46. Uncertainty: Introduction and Quantification Malte Helmert and Gabriele R oger University of Basel May 24, 2017 Introduction Probability Theory Inference from Full Joint Distributions Bayes

432 views • 29 slides
CHiME Challenge: Approaches to Robustness using Beamforming and Uncertainty-of-Observation

CHiME Challenge: Approaches to Robustness using Beamforming and Uncertainty-of-Observation

Department of Electrical Engineering and Information Sciences INESC-ID Lisboa Institute of Communication Acoustics (IKA) CHiME Challenge: Approaches to Robustness using Beamforming and Uncertainty-of-Observation Techniques Dorothea Kolossa 1 ,

874 views • 32 slides
Uncertainty quantification in computer experiments with polynomial chaos J. KO 1 with J. GARNIER 2

Uncertainty quantification in computer experiments with polynomial chaos J. KO 1 with J. GARNIER 2

Uncertainty quantification in computer experiments with polynomial chaos J. KO 1 with J. GARNIER 2 , D. LUCOR 3 & A. DIPANKAR 4 1. j o r d a n . k o @ m a c . c o m 2. Laboratoire de Probabilit es et Mod` eles Al eatoires, Universit

1.08k views • 52 slides
Robust nonlinear Optimization Maren Mahsereci Workshop on Uncertainty Quantification 09 / 15 /

Robust nonlinear Optimization Maren Mahsereci Workshop on Uncertainty Quantification 09 / 15 /

Robust nonlinear Optimization Maren Mahsereci Workshop on Uncertainty Quantification 09 / 15 / 2016, Sheffield Emmy Noether Group on Probabilistic Numerics Department of Empirical Inference Max Planck Institute for Intelligent Systems

889 views • 55 slides
IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L.

IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L.

IGA-based Multi-Index Stochastic Collocation for Uncertainty Quantification J. Beck 1 , L. Tamellini 2 , R. Tempone 1 , 3 1 King Abdullah University of Science and Technology, Saudi Arabia 2 CNR-IMATI, Italy 3 Alexander von Humboldt Professor in

691 views • 65 slides
Uncertainty quantification in complex systems: emerging diseases,predicting science and the

Uncertainty quantification in complex systems: emerging diseases,predicting science and the

Uncertainty quantification in complex systems: emerging diseases,predicting science and the functional structure of the brain Lus M. A. Bettencourt Los Alamos National Laboratory & Santa Fe Institute http://math.lanl.gov/~lmbett

937 views • 67 slides
A validation/uncertainty quantification analysis for a 1.5 MW oxy-coal fired L1500 furnace using a

A validation/uncertainty quantification analysis for a 1.5 MW oxy-coal fired L1500 furnace using a

A validation/uncertainty quantification analysis for a 1.5 MW oxy-coal fired L1500 furnace using a swirling boundary condition Oscar H. D az-Ibarra, Jennifer Spinti, Andrew Fry, Benjamin Isaac, Jeremy N. Thornock, Michal Hradisky, Sean

562 views • 39 slides

More recommend