Deep Learning With Constraints Yatin Nandwani Work done in - PowerPoint PPT Presentation

Deep Learning With Constraints Yatin Nandwani Work done in collaboration with Abhishek Pathak Under the guidance of Prof. Mausam and Prof. Parag Singla

Learning with Constraints: Motivation ➔ Modern day AI == Deep Learning (DL) [Learn from Data] 2

Learning with Constraints: Motivation ➔ Modern day AI == Deep Learning (DL) [Learn from Data] ➔ Can we inject symbolic knowledge in Deep Learning? E.g. Person => Noun [Learn from Data Knowledge] (credit: Vivek S Kumar) 3

Learning with Constraints: Motivation ➔ Modern day AI == Deep Learning (DL) [Learn from Data] ➔ Can we inject symbolic knowledge in Deep Learning? E.g. Person => Noun [Learn from Data Knowledge] (credit: Vivek S Kumar) ➔ Constraints: One of the ways of representing symbolic knowledge. 4

Learning with Constraints: Motivation ➔ Modern day AI == Deep Learning (DL) [Learn from Data] ➔ Can we inject symbolic knowledge in Deep Learning? E.g. Person => Noun [Learn from Data Knowledge] (credit: Vivek S Kumar) ➔ Constraints: One of the ways of representing symbolic knowledge. ➔ Limited work in training DL models with (soft) constraints 5

Learning with Constraints: Motivation ➔ Modern day AI == Deep Learning (DL) [Learn from Data] ➔ Can we inject symbolic knowledge in Deep Learning? E.g. Person => Noun [Learn from Data Knowledge] (credit: Vivek S Kumar) ➔ Constraints: One of the ways of representing symbolic knowledge. ➔ Limited work in training DL models with (soft) constraints ➔ What if constraints are hard? 6

Neural + Constraints ❖ Augmenting deep neural models ( DNN ) with Domain Knowledge ( DK ) ❖ Domain Knowledge expressed in the form of Constraints ( C ) ➢ Learning with (hard) constraints: Learn DNN weights s.t. output satisfies constraints C 7

Related Work

Learning with Constraints: Running Example ● Task: Fine Grained Entity Typing 15

Learning with Constraints: Running Example Input: Bag of Mentions Sample Mention: “ Barack Obama is the President of the United States ” Output: president, leader, politician... 16

Learning with Constraints: Running Example Input: Bag of Mentions Sample Mention: “ Barack Obama is the President of the United States ” Output: president, leader, politician... 17

Learning with Constraints: Running Example ● Constraints: Hierarchy on Output label space 18

Learning with Constraints: Running Example ● Constraints: Hierarchy on Output label space Person Artist Lawyer Doctor Musician Actor 19

Learning with Constraints: Running Example ● Constraints: Hierarchy on Output label space Person Source: Artist Lawyer Doctor https://github.com/iesl/TypeNet https://github.com/MurtyShikhar/Hierarchical-Typing Musician Actor 20

Learning with Constraints: Representation of Constraints ➔ Using Soft Logic 21

Learning with Constraints: Representation of Constraints Equivalently: 25

Learning with Constraints: Representation of Constraints Define: k th Constraint i th Data point Inequality Constraint: 28

Learning with Constraints: Formulation Unconstrained Problem : Any standard loss function, say Cross Entropy 29

Learning with Constraints: Formulation Unconstrained Problem : Any standard loss function, say Cross Entropy Constrained Problem 30

Learning with Constraints: Formulation Constrained Problem Where: m: Size of training data K: Number of Constraints 31

Learning with Constraints: Formulation Constrained Problem Lagrangian

Learning with Constraints: Formulation Constrained Problem Lagrangian Primal Dual

Learning with Constraints: Formulation Constrained Problem Where: Issue: O(mK) #constraints m: Size of training data K: Number of Constraints i.e. mK Lagrange Multipliers! 34

Learning with Constraints: Reduce # Constraints H(c) 35

Learning with Constraints: Reduce # Constraints H(c) Equivalent 36

Learning with Constraints: Reduce # Constraints H(c) Equivalent 37

Learning with Constraints: Reduce # Constraints Originally: 38

Learning with Constraints: Reduce # Constraints Originally: Now: Define: 39

Learning with Constraints: Reduce # Constraints Originally: Now: O(K) #constraints Define: 40

Learning with Constraints: Primal-Dual Formulation Lagrangian 41

Learning with Constraints: Primal-Dual Formulation Lagrangian Primal Dual 42

Learning with Constraints: Parameter Update 43

Learning with Constraints: Training Algorithm 48

Learning with Constraints: Training Algorithm Crucial for convergence guarantees! 52

Learning with Constraints: Experiments Typenet MAP Scores Constraint Violations 5% 10% 100% 5% 10% 100% Scenario Data Data Data Data Data Data B 68.6 22,715 B+H 68.71 22,928 B+C B+S 53

Learning with Constraints: Experiments Typenet MAP Scores Constraint Violations 5% 10% 100% 5% 10% 100% Scenario Data Data Data Data Data Data B 68.6 22,715 B+H 68.71 22,928 B+C 80.13 25 B+S 82.22 41 54

Learning with Constraints: Experiments Typenet MAP Scores Constraint Violations 5% 10% 100% 5% 10% 100% Scenario Data Data Data Data Data Data B 68.6 69.2 70.5 22,715 21,451 22,359 B+H 68.71 69.31 71.77 22,928 21,157 24,650 B+C 80.13 81.36 82.80 25 45 12 B+S 82.22 83.81 41 26 55

Learning with Constraints: Experiments NER Task: Named Entity Recognition Auxiliary Task: Part of Speech Tagging 56

Learning with Constraints: Experiments NER Task: Named Entity Recognition Auxiliary Task: Part of Speech Tagging Architecture: Common LSTM encoder and task specific classifier 57

Learning with Constraints: Experiments NER Task: Named Entity Recognition Auxiliary Task: Part of Speech Tagging Architecture: Common LSTM encoder and task specific classifier Constraints: 16 constraints of type: Person => Noun 58

Learning with Constraints: Experiments NER 59

Learning with Constraints: Experiments SRL Task: Semantic Role Labelling Auxiliary Info: Syntactic Parse Trees 60

Learning with Constraints: Experiments SRL • For each clause, determine the semantic role played by each noun phrase that is an argument to the verb. agent patient source destination instrument – John drove Mary from Austin to Dallas in his Toyota Prius. – The hammer broke the window. •Also referred to a “case role analysis,” “thematic analysis,” and “shallow semantic parsing” Slide Credit: Ray Mooney

Learning with Constraints: Experiments SRL Task: Semantic Role Labelling Auxiliary Info: Syntactic Parse Trees Architecture: State-of-the-art based on ELMo embeddings 62

Learning with Constraints: Experiments SRL Task: Semantic Role Labelling Auxiliary Info: Syntactic Parse Trees Architecture: State-of-the-art based on ELMo embeddings Constraints: Transition Constraints & span constraints 63

Learning with Constraints: Experiments SRL Constraints: Transition Constraints e.g. B-Arg( i ) => I- Arg( i+1 ) Span Constraints: Semantic spans should be subset of syntactic spans 64

Learning with Constraints: Experiments SRL: Syntactic Parse Tree for span constraints Slide Credit: Ray Mooney

Learning with Constraints: Experiments SRL F1 Score Total Constraint Violations Scenario 1% Data 5% Data 10% Data 1% Data 5% Data 10% Data B 62.99 14,857 CL 66.21 9,406 B+CI CL + CI 66

Learning with Constraints: Experiments SRL F1 Score Total Constraint Violations Scenario 1% Data 5% Data 10% Data 1% Data 5% Data 10% Data B 62.99 72.64 76.04 14,857 9,708 7,704 CL 66.21 74.27 77.19 9,406 7,461 5,836 B+CI CL + CI 67

Learning with Constraints: Experiments SRL F1 Score Total Constraint Violations Scenario 1% Data 5% Data 10% Data 1% Data 5% Data 10% Data B 62.99 72.64 76.04 14,857 9,708 7,704 CL 66.21 74.27 77.19 9,406 7,461 5,836 B+CI 67.9 75.96 78.63 5,737 4,247 3,654 CL + CI 68.71 76.51 78.72 5,039 3,963 3,476 68

Reviews Doubt 1. Why constraint violations even though they are hard.

Deep Learning With Constraints Yatin Nandwani Work done in - PowerPoint PPT Presentation

Deep Learning With Constraints Yatin Nandwani Work done in collaboration with Abhishek Pathak Under the guidance of Prof. Mausam and Prof. Parag Singla Learning with Constraints: Motivation Modern day AI == Deep Learning (DL) [Learn from

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Deep learning for NLP: Introduction CS 6956: Deep Learning for NLP Words are a very fantastical

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Bayesian Deep Learning Mohd Adnan Problems With Deep Learning What does a model not know?

Deep Learning on Massively Parallel Processing Databases Frank McQuillan Feb 2019 2 A Brief

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

DSSTNE: Deep Learning At Scale For Large Sparse Datasets

Deep Learning With Differential Privacy Presenter: Xiaojun Xu Deep Learning Framework

Imposing Hard Constraints on Deep Networks: Promises and Limitations min w R ( w ) s.t C ( w ) =