Learning Logistic Circuits Yitao Liang, Guy Van den Broeck January - PowerPoint PPT Presentation

Mar 06, 2024 •439 likes •650 views

Learning Logistic Circuits Yitao Liang, Guy Van den Broeck January 31, 2019 Which model to choose Neural Networks: Classical AI Methods: Hungry? $25? Sleep? Restaurant? Black Box Clear Modeling Assumption Good performance on

Learning Logistic Circuits Yitao Liang, Guy Van den Broeck January 31, 2019
Which model to choose Neural Networks: Classical AI Methods: Hungry? $25? Sleep? Restaurant? … “Black Box” Clear Modeling Assumption Good performance on Image Classification 1
Starting Point: Probabilistic Circuits A promising synthesis of the two SPN State-of-the-art on Density Estimation !"($) 3 2
What if we only want to learn a classifier !" # $) 4 3
Logical Circuits 1 0 1 Input: 1 1 A ¬ A 0 1 0 1 0 A B C D 0 = 1 AND 0 0 1 1 0 1 0 1 1 0 0 0 1 Bottom-up Evaluation B ¬ B 1 0 C ¬ C ¬ D D 1 0 1 0 4
Logical -> Probabilistic Circuits 0 . 9 0 . 1 A ¬ A Red Parameters: 0 . 2 0 . 8 0 . 4 0 . 6 Conditional Probabilities 0 . 1 0 . 9 0 . 3 0 . 7 0 . 9 0 . 8 0 . 1 0 . 2 B ¬ B C ¬ C ¬ D D 5
Logical -> Probabilistic Circuits !"($, &, ', () = +. +-. Input: 0 . 9 0 . 1 0 0.096 Pr( A, B, C, D ) A B C D A ¬ A 0.096 0.194 1 0 0 . 2 0 . 8 0 . 4 0 . 6 0.24= 0.8*0.3 0 1 1 0 ? 0.01 0.24 0.00 0.1= 0.1*1 + 0.9*0 0.0 0.3 0.1 0.8 0 . 1 0 . 9 0 . 3 0 . 7 Multiply the parameters 0 . 9 0 . 8 0 . 1 0 . 2 0 0 0 1 bottom-up B ¬ B 1 0 C ¬ C ¬ D D 1 0 1 0 6
Evaluate Logistic Circuits % !" # = % &, (, ), *) = %,-./(1%.3) = 4. 563 − 2 . 6 − 5 . 8 Input: A ¬ A 0 1 Pr( Y | A, B, C, D ) A B C D − 1 3 2 . 3 4 0 1 1 0 ? − 0 . 5 0 . 3 1 . 5 2 . 8 Multiply the parameters − 4 1 3 . 9 4 bottom-up B ¬ B 1 0 Logistic function on final output C ¬ C ¬ D D 1 0 1 0 7
Are logistic circuits amenable to tractable learning 8
Special Case: Logistic Regression Logistic Regression 1 Pr # = 1 &, (, ), * = ) 1 + ex p( − & ∗ 3 4 − ¬& ∗ 3 ¬4 − ( ∗ 3 6 − ⋯ What about other logistic circuits in more general forms? 9
Parameter Learning Pr($ = 1 ∣ ( = 0, + = 1, , = 1, - = 0) “Hot” wires are active features 10
Parameter Learning Due to decomposability and determinism, reduce to logistic regression Features associated with each wire “Global Circuit Flow” Convex Parameter learning 11
Structure Learning Generate Calculate candidate Variance operations Execute the best Similar to LearnPsdd Split nodes to reduce variance of gradients 14 12
Comparable Accuracy with Neural Nets 15 13
Significantly Smaller in Size 16 14
Better Data Efficiency 17 15
Probabilistic -> Logistic Circuits Probabilities become log-odds Discriminative Counterparts 18 16
What do Features Mean This is the feature that contributes the most to this image’s classification probability feature value : 0.925 feature weight : 3.489 feature interpretation: curvy lines and hallow center 19 17
Conclusion Logistic circuits : • Synthesis of symbolic AI and statistical learning • Discriminative counterparts of probabilistic circuits • Convex parameter learning • Simple heuristic for structure learning • Good performance • Easy to interpret 18
Thanks https://github.com/UCLA-StarAI/LogisticCircuit

Recommend

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den Broeck KULeuven Symposium Dec 12, 2018 Outline Learning Adding knowledge to deep learning Logistic circuits for image classification

782 views • 49 slides

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den Broeck Stanford Nov 14, 2018 Foundation: Logical Circuit Languages Negation Normal Form Circuits = (sun rain rainbow) [Darwiche 2002]

890 views • 75 slides

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den Broeck HRL/ACTIONS @ KR Oct 28, 2018 Foundation: Logical Circuit Languages Negation Normal Form Circuits = (sun rain rainbow)

1.01k views • 74 slides

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den

Probabilistic and Logistic Circuits: A New Synthesis of Logic and Machine Learning Guy Van den Broeck RelationalAI ArrowCon Feb 5, 2019 Which method to choose? Classical AI Methods: Neural Networks: Hungry $25? ? Restaura Sleep? nt?

819 views • 56 slides

w o o o o o o o x o o o x o o that represents how aligned the o x x x x x x

Outline Logistic function IAML: Logistic Regression Logistic regression Learning logistic regression Optimization Nigel Goddard and Victor Lavrenko The power of non-linear basis functions School of Informatics

338 views • 6 slides

w o o o o o o o x o o o x o o o that represents how aligned the x x x x x x

Outline Logistic function IAML: Logistic Regression Logistic regression Learning logistic regression Optimization Nigel Goddard The power of non-linear basis functions School of Informatics Least-squares classification

622 views • 6 slides

w o o o o o o o o x o o o o o x o that represents how aligned the o x x x x

Outline Logistic function Logistic regression IAML: Logistic Regression Learning logistic regression Optimization Charles Sutton and Victor Lavrenko The power of non-linear basis functions School of Informatics

343 views • 6 slides

1 Relation Between Multinomial Logistic Regression Nave Bayes and Logistic Regression

Logistic Regression Assumes a parametric form for directly estimating P( Y | X ). For binary concepts, this is: CS 391L: Machine Learning: 1 P Y = X = ( 1 | ) n + w + w i X Bayesian Learning: 1 exp( ) = i 0 i 1

308 views • 5 slides

LEARNING Outline Math Behind Logistic Regression Visualizing Logistic Regression Loss

Logistic Regression CSCI 447/547 MACHINE LEARNING Outline Math Behind Logistic Regression Visualizing Logistic Regression Loss Function Minimizing Log Likelihood Function Batch/Full Logistic Regression Gradient Descent for

498 views • 29 slides

Machine Learning Logistic Regression Hamid R. Rabiee Spring 2015

Machine Learning Logistic Regression Hamid R. Rabiee Spring 2015 http://ce.sharif.edu/courses/93-94/2/ce717-1 / Agenda Agenda Probabilistic Classification Introduction to Logistic regression Binary logistic regression Logistic

362 views • 20 slides

Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt

Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Logistic Regression James H. Steiger

777 views • 54 slides

Logistic Regression Machine Learning 1 Where are we? We have seen the following ideas

Logistic Regression Machine Learning 1 Where are we? We have seen the following ideas Linear models Learning as loss minimization Bayesian learning criteria (MAP and MLE estimation) The Nave Bayes classifier 2 This lecture

822 views • 63 slides

Introduction to Machine Learning Classification: Logistic Regression

Introduction to Machine Learning Classification: Logistic Regression compstat-lmu.github.io/lecture_i2ml MOTIVATION A discriminant approach for directly modeling the posterior probabilities ( x | ) of the labels is logistic regression . For

484 views • 9 slides

Binary Logistic Regression + Multinomial Logistic Regression Matt Gormley Lecture 10 Feb. 17,

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Binary Logistic Regression + Multinomial Logistic Regression Matt Gormley Lecture 10 Feb. 17, 2020 1 Reminders

452 views • 43 slides

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression Gradient Descent M. Magdon-Ismail CSCI 4100/6100 recap: Linear Classification and Regression The linear signal: s = w t x Good Features are Important

272 views • 23 slides

From Logistic Regression to Neural Networks CMSC 470 Marine Carpuat Logistic Regression What

From Logistic Regression to Neural Networks CMSC 470 Marine Carpuat Logistic Regression What you should know How to make a prediction with logistic regression classifier How to train a logistic regression classifier Machine learning

502 views • 33 slides

Reasoning and Learning Guy Van den Broeck UC Berkeley EECS Feb 11, 2019 Outline: Reasoning

Towards a New Synthesis of Reasoning and Learning Guy Van den Broeck UC Berkeley EECS Feb 11, 2019 Outline: Reasoning Learning 1. Deep Learning with Symbolic Knowledge 2. Probabilistic and Logistic Circuits 3. High-Level Probabilistic

779 views • 63 slides

Applied Machine Learning Logistic and Softmax Regression Siamak Ravanbakhsh COMP 551 (Fall 2020)

Applied Machine Learning Logistic and Softmax Regression Siamak Ravanbakhsh COMP 551 (Fall 2020) Learning objectives what are linear classifiers logistic regression model loss function maximum likelihood view multi-class classification

721 views • 29 slides

Machine Learning - MT 2016 8. Classification: Logistic Regression Varun Kanade University of

Machine Learning - MT 2016 8. Classification: Logistic Regression Varun Kanade University of Oxford November 2, 2016 Logistic Regression Logistic Regression is actually a classification method In its simplest form it is a binary (two classes)

486 views • 14 slides

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Logistic Regression 2- class logistic

308 views • 15 slides

Logistic Regression CS60010: Deep Learning Abir Das IIT Kharagpur Jan 22, 23 and 24, 2020

Logistic Regression CS60010: Deep Learning Abir Das IIT Kharagpur Jan 22, 23 and 24, 2020 Logistics Agenda Linear Regression Logistic Regression Some Logistics Related Information This Friday (Jan 24), no paper will be presented. It will

933 views • 50 slides

Todays lecture Logistic regression How can we use logistic regression for reranking? Shay

Todays lecture Logistic regression How can we use logistic regression for reranking? Shay Cohen How do we set the parameters of a logistic regression model? (based on slides by Sharon Goldwater) How is logistic regression related

652 views • 9 slides

L101: Optimization fundamentals Previous lecture Logistic regression parameter learning:

L101: Optimization fundamentals Previous lecture Logistic regression parameter learning: Supervised machine learning algorithms typically involve optimizing a loss over the training data: This is an instance of numerical optimization , i.e.

486 views • 21 slides

Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical

MACHINE LEARNING TOOLBOX Logistic regression on Sonar Machine Learning Toolbox Classification models Categorical (i.e. qualitative) target variable Example: will a loan default? Still a form of supervised learning Use a

315 views • 27 slides