Numerical Gaussian Processes (Physics Informed Learning Machines) Maziar Raissi Division of Applied Mathematics, Brown University, Providence, RI, USA maziar_raissi@brown.edu June 7, 2017
Probabilistic Numerics v.s. Numerical Gaussian Processes 1 Probabilistic numerics aim to capitalize on the recent developments in probabilistic machine learning to revisit classical methods in numerical analysis and mathematical physics from a statistical inference point of view. Maziar Raissi | Numerical Gaussian Processes
Probabilistic Numerics v.s. Numerical Gaussian Processes 1 Probabilistic numerics aim to capitalize on the recent developments in probabilistic machine learning to revisit classical methods in numerical analysis and mathematical physics from a statistical inference point of view. This is exciting. However, it would be even more exciting if we could do the exact opposite. Maziar Raissi | Numerical Gaussian Processes
Probabilistic Numerics v.s. Numerical Gaussian Processes 1 Probabilistic numerics aim to capitalize on the recent developments in probabilistic machine learning to revisit classical methods in numerical analysis and mathematical physics from a statistical inference point of view. This is exciting. However, it would be even more exciting if we could do the exact opposite. Numerical Gaussian processes aim to capitalize on the long-standing developments of classical methods in numerical analysis and revisits machine leaning from a mathematical physics point of view. Maziar Raissi | Numerical Gaussian Processes
Physics Informed Learning Machines 2 Numerical Gaussian processes enable the construction of data-efficient learning machines that can encode physical conservation laws as structured prior information. Maziar Raissi | Numerical Gaussian Processes
Physics Informed Learning Machines 2 Numerical Gaussian processes enable the construction of data-efficient learning machines that can encode physical conservation laws as structured prior information. Numerical Gaussian processes are essentially physics informed learning machines. Maziar Raissi | Numerical Gaussian Processes
Content 3 Motivating Example Introduction to Gaussian Processes Prior Training Posterior Numerical Gaussian Processes Burgers’ Equation – Nonlinear PDEs Backward Euler Prior Training Posterior General Framework Linear Multi-step Methods Runge-Kutta Methods Experiments Navier-Stokes Equations Maziar Raissi | Numerical Gaussian Processes
3 Motivating Example Maziar Raissi | Numerical Gaussian Processes
Road Networks 4 Consider a 2 × 2 junction as shown below. 2 1 3 4 Roads have length L i , i = 1 , 2 , 3 , 4. Maziar Raissi | Numerical Gaussian Processes
Road Networks 5 Maziar Raissi | Numerical Gaussian Processes
Hyperbolic Conservation Law 6 The road traffic densities ρ i ( t , x ) ∈ [ 0 , 1 ] satisfy the one-dimensional hyperbolic conservation law ∂ t ρ i + ∂ x f ( ρ i ) = 0 , on [ 0 , T ] × [ 0 , L i ] . Here, f ( ρ ) = ρ ( 1 − ρ ) . Maziar Raissi | Numerical Gaussian Processes
Black Box Initial Conditions 7 The densities must satisfy the initial conditions ρ i ( 0 , x ) = ρ 0 i ( x ) , where ρ 0 i ( x ) are black-box functions. This means that ρ 0 i ( x ) are observable only through noisy measurements { x 0 i , ρ 0 i } . Maziar Raissi | Numerical Gaussian Processes
7 Introduction to Gaussian Processes Maziar Raissi | Numerical Gaussian Processes
Gaussian Processes 8 A Gaussian process f ( x ) ∼ GP ( 0 , k ( x , x ′ ; θ )) , is just a shorthand notation for � f ( x ) � 0 � k ( x , x ; θ ) � � � k ( x , x ′ ; θ ) ∼ N ( , . f ( x ′ ) 0 k ( x ′ , x ; θ ) k ( x ′ , x ′ ; θ ) Maziar Raissi | Numerical Gaussian Processes
Gaussian Processes 9 A Gaussian process f ( x ) ∼ GP ( 0 , k ( x , x ′ ; θ )) , is just a shorthand notation for � f ( x ) � 0 � k ( x , x ; θ ) � � � k ( x , x ′ ; θ ) ∼ N ( , . f ( x ′ ) 0 k ( x ′ , x ; θ ) k ( x ′ , x ′ ; θ ) Maziar Raissi | Numerical Gaussian Processes
Squared Exponential Covariance Function 10 A typical example for the kernel k ( x , x ′ ; θ ) is the squared exponential covariance function, i.e., � � − 1 k ( x , x ′ ; θ ) = γ 2 exp 2 w 2 ( x − x ′ ) 2 , where θ = ( γ, w ) are the hyper-parameters of the kernel. Maziar Raissi | Numerical Gaussian Processes
Training 11 Given a dataset { x , y } of size N , the hyper-parameters θ and the noise variance parameter σ 2 can be trained by minimizing the negative log marginal likelihood NLML ( θ, σ ) = 1 2 y T K − 1 y + 1 2 log | K | + N 2 log ( 2 π ) , resulting from y ∼ N ( 0 , K ) , where K = k ( x , x ; θ ) + σ 2 I . Maziar Raissi | Numerical Gaussian Processes
Prediction 12 Having trained the hyper-parameters and parameters of the model, one can use the posterior distribution f ( x ∗ ) | y ∼ N ( k ( x ∗ , x ) K − 1 y , k ( x ∗ , x ∗ ) − k ( x ∗ , x ) K − 1 k ( x , x ∗ )) . to make predictions at a new test point x ∗ . Maziar Raissi | Numerical Gaussian Processes
Prediction 12 Having trained the hyper-parameters and parameters of the model, one can use the posterior distribution f ( x ∗ ) | y ∼ N ( k ( x ∗ , x ) K − 1 y , k ( x ∗ , x ∗ ) − k ( x ∗ , x ) K − 1 k ( x , x ∗ )) . to make predictions at a new test point x ∗ . This is obtained by writing the joint distribution � f ( x ∗ ) � 0 � k ( x ∗ , x ) � � � k ( x ∗ , x ) ∼ N ( , . y 0 k ( x , x ∗ ) K Maziar Raissi | Numerical Gaussian Processes
Example Code 13 Maziar Raissi | Numerical Gaussian Processes
13 Numerical Gaussian Processes Maziar Raissi | Numerical Gaussian Processes
Numerical Gaussian Processes Definition 14 Numerical Gaussian processes are Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Maziar Raissi | Numerical Gaussian Processes
Example: Burgers’ Equation 15 Burgers’ equation is a fundamental non-linear partial differential equation arising in various areas of applied mathematics, including fluid mechanics, nonlinear acoustics, gas dynamics, and traffic flow. Maziar Raissi | Numerical Gaussian Processes
Example: Burgers’ Equation 15 Burgers’ equation is a fundamental non-linear partial differential equation arising in various areas of applied mathematics, including fluid mechanics, nonlinear acoustics, gas dynamics, and traffic flow. In one space dimension the Burgers’ equation reads as u t + uu x = ν u xx , along with Dirichlet boundary conditions u ( t , − 1 ) = u ( t , 1 ) = 0 , where u ( t , x ) denotes the unknown solution and ν = 0 . 01 /π is a viscosity parameter. Maziar Raissi | Numerical Gaussian Processes
Problem Setup Burgers’ Equation 16 Let us assume that all we observe are noisy measurements { x 0 , u 0 } of the black-box initial function u ( 0 , x ) . Given such measurements, we would like to solve the Burgers’ equation while propagating through time the uncertainty associated with the noisy initial data. Maziar Raissi | Numerical Gaussian Processes
Burgers’ equation Movie Code 17 Maziar Raissi | Numerical Gaussian Processes
Burgers’ equation 18 It is remarkable that the proposed methodology can effectively propagate an infinite collection of correlated Gaussian random variables (i.e., a Gaussian process) through the complex nonlinear dynamics of the Burgers’ equation. Maziar Raissi | Numerical Gaussian Processes
Backward Euler Burgers’ Equation 19 Let us apply the backward Euler scheme to the Burgers’ equation. This can be written as dx u n − ν ∆ t d 2 u n + ∆ tu n d dx 2 u n = u n − 1 . Maziar Raissi | Numerical Gaussian Processes
Backward Euler Burgers’ Equation 20 Let us apply the backward Euler scheme to the Burgers’ equation. This can be written as dx u n − ν ∆ t d 2 u n + ∆ t µ n − 1 d dx 2 u n = u n − 1 . Maziar Raissi | Numerical Gaussian Processes
Prior Assumption Burger’s Equation 21 Let us make the prior assumption that u n ( x ) ∼ GP ( 0 , k ( x , x ′ ; θ )) , is a Gaussian process with a neural network covariance function 2 ( σ 2 0 + σ 2 xx ′ ) k ( x , x ′ ; θ ) = 2 π sin − 1 , � � σ 2 0 + σ 2 x 2 ) � � σ 2 0 + σ 2 x ′ 2 ) � ( 1 + 2 ( 1 + 2 � σ 2 0 , σ 2 � where θ = denotes the hyper-parameters. Maziar Raissi | Numerical Gaussian Processes
Numerical Gaussian Process Burgers’ Equation – Backward Euler 22 This enables us to obtain the following Numerical Gaussian Process � � �� k n , n k n , n − 1 u n � � u , u u , u ∼ GP 0 , . u n − 1 k n − 1 , n − 1 u , u Maziar Raissi | Numerical Gaussian Processes
Kernels Burgers’ Equation – Backward Euler 23 The covariance functions for the Burgers’ equation example are given by k n , n u , u = k , dx ′ k − ν ∆ t d 2 k + ∆ t µ n − 1 ( x ′ ) d k n , n − 1 = dx ′ 2 k . u , u Maziar Raissi | Numerical Gaussian Processes
Recommend
More recommend
