Understanding and mitigating gradient flow pathologies in - PowerPoint PPT Presentation

Understanding and mitigating gradient flow pathologies in physics-informed neural networks Paris Perdikaris Sifan Wang Department of Mechanical Engineering Applied Mathematics & Computational Science University of Pennsylvania University of Pennsylvania email: pgp@seas.upenn.edu email: sifanw@sas.upenn.edu Supported by: ICERM Computational Statistics and Data-Driven Models April 21, 2020

DuHicbVJb 9MwFE5WLqPcNnjkxWJC2kSpknbAmFQxDSR4QKhM7CI1WXCck9Sa7US+oBXLv4/fsH+D2xVEth3Jysl3jr9z+Zw3jCodR fhSufW7Tt3V+917z94+Ojx2vqTI1UbSeCQ1KyWJzlWwKiAQ0 1g5NGAuY5g+P87M 8fvwTpK 1+K5nDaQcV4KWlGDtoWztIuFYTwlm9ovbTPQUN 5CuyOUGFGAzCUmYJPSf2zs7NfMuEQZnjDKqVaZpaPYnS5gNElybo3L6Ksys5dEntBj5x7bSk8HLks0nGv7EWuMSqodenlzlYT59gvs/nV24CbXOLfSv3zj6UxRopCEyjAs6a/FZA5laxtRP1oYu 7ES2cjWNo4Ww9/J0VNDAehCcNKTeKo0anFUlPCwHUTo6DB5AxXMPGuwBxUahcaOPTCIwUqa+mP0GiB/n/DYq7UjOc+cz6XuhqbgzfFJkaXO6mlojEaBLksVBqGdI3mgqKCSiCazbyDiaS+V0Sm2O9Se9lbVXLemsESLAiw9lx4u8ENyJ6n15iN7BsqemiHCtfjWFZUjK +/+umtoKag5azNqWRzKFuC5pPpOuaqXYmZlXtu53ygd9sNymg9C94sTU7BiEOoHD24NO+s/HrYS/qRTcl7TMDy6woinvR0J93set6 eOrQl93jgb9eNgf Nve2NtfPoLV4FnwPNgM4uBtsBd8DsbBYUDC9yGEIqw7u50fnapDL1NXwuWdp0HLOvIP9X86Q = </latexit> <latexit sha1_base64="1HGZtmjbtzdRdyLbi+i+HT/d8uc=">A Physics of AI: Two schools of thought 1. Physics is implicitly 1 baked in specialized 3 2 neural architectures with 4 5 6 strong inductive biases 1 (e.g. invariance to simple 3 2 group symmetries). 4 5 7 6 *figures from Kondor, R., Son, H. T., Pan, H., Anderson, B., & Trivedi, S. (2018). Covariant compositional networks for learning graphs. arXiv preprint arXiv:1801.02144. PDE( λ ) ∂ T f ∂ t NN( x, t ; θ ) ∂ t − λ ∂ 2 ˆ 2. Physics is explicitly ∂ ˆ u u ∂ x 2 � � ∂ 2 imposed by constraining ∂ x 2 the output of conventional x � � u ˆ Minimize neural architectures with . . . . u ( x, t ) − g D ( x, t ) ˆ I Loss θ ∗ t . . weak inductive biases. T b � � ∂ ∂ ˆ u ∂ n ( x, t ) − g R ( u, x, t ) Psichogios & Ungar, 1992 ∂ n Lagaris et. al., 1998 BC & IC Raissi et. al., 2019 N u 1 1 X Lu et. al., 2019 [ u i − f θ ( x i )] 2 L ( θ ) := + λ R [ f θ ( x )] Zhu et. al., 2019 N u i =1 | {z } | {z } Physics regularization Data fit

Physics-informed Neural Networks ⊂ @ 2 u @ 2 u ✓ x ; @u , . . . , @u ◆ x ∈ Ω , f ; , . . . , ; . . . ; λ = 0 , B ( u, x ) = 0 on @ Ω , @x 1 @x d @x 1 @x 1 @x 1 @x d PDE( λ ) ∂ T f ∂ t NN( x, t ; θ ) ∂ t − λ ∂ 2 ˆ ∂ ˆ u u ∂ x 2 ∂ 2 � � ∂ x 2 x � � u ˆ Minimize . . . . u ( x, t ) − g D ( x, t ) ˆ I Loss θ ∗ t . . T b � � ∂ ∂ ˆ u ∂ n ( x, t ) − g R ( u, x, t ) ∂ n BC & IC Automatic differentiation Psichogios, D. C., & Ungar, L. H. (1992). A hybrid neural network - first principles approach to process modeling. AIChE Journal, 38(10), 1499-1511. Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998). Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5), 987-1000. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707. Lu, L., Meng, X., Mao, Z., & Karniadakis, G. E. (2019). DeepXDE: A deep learning library for solving differential equations. arXiv preprint arXiv: 1907.04502.