from maxent to machine learning and back
play

From Maxent to Machine Learning and Back T. Sears ANU March 2007 - PowerPoint PPT Presentation

Dec 24, 2023 •262 likes •1.24k views

From Maxent to Machine Learning and Back T. Sears ANU March 2007 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 1 / 36 50 Years Ago . . . The principles and mathematical methods of statistical mechanics are seen to

  1. From Maxent to Machine Learning and Back T. Sears ANU March 2007 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 1 / 36

  2. 50 Years Ago . . . ”The principles and mathematical methods of statistical mechanics are seen to be of much more general applicability. . . In the problem of prediction, the maximization of entropy is not an application of a law of physics, but merely a method of reasoning which ensures that no unconscious arbitrary assumptions have been introduced.” –E.T. Jaynes, 1957 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 2 / 36

  3. “. . . a method of reasoning . . . ” Jenkins, if I want another yes-man I’ll build one.

  4. Outline Generalizing Maxent 1 Two Examples 2 Broader Comparisons 3 Extensions/Conclusions 4 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 4 / 36

  5. Generalizing Maxent You are here Generalizing Maxent 1 Two Examples 2 Broader Comparisons 3 Extensions/Conclusions 4 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 5 / 36

  6. Generalizing Maxent The Classic Maxent Problem Minimize negative entropy subject to linear constraints: A is M × N . N � M < N , a ”wide” matrix. min p S ( p ) := p i log( p i ) b is a data vector. i =1 � B � subject to A := T 1 Ap = b contains a normalization p i ≥ 0 constraint. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 6 / 36

  7. Generalizing Maxent Extending the Classic Maxent Problem min p S ( p ) subject to Ap = b Original problem. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  8. Generalizing Maxent Extending the Classic Maxent Problem min p S ( p ) + δ { 0 } ( Ap − b ) Original problem. Convert constraints to a convex function. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  9. Generalizing Maxent Extending the Classic Maxent Problem min p S ( p ) + δ { 0 } ( || Ap − b || P ) Original problem. Convert constraints to a convex function. Use any norm... T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  10. Generalizing Maxent Extending the Classic Maxent Problem min p S ( p ) + δ ǫ B P ( Ap − b ) Original problem. Convert constraints to a convex function. Use any norm... and relax constraints. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  11. Generalizing Maxent Extending the Classic Maxent Problem min p ∆ F ( p , p 0 ) + δ ǫ B P ( Ap − b ) Original problem. Convert constraints to a convex function. Use any norm... and relax constraints. Generalize SBG entropy to Bregman divergence. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  12. Generalizing Maxent Extending the Classic Maxent Problem µ F ∗ ( A T µ + p ∗ min 0 ) + � µ , b � + ǫ || µ || Q Original problem. Convert constraints to a convex function. Use any norm... and relax constraints. Generalize SBG entropy to Bregman divergence. Find the Fenchel dual problem to solve. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  13. Generalizing Maxent Extending the Classic Maxent Problem µ F ∗ ( A T µ + p ∗ min 0 ) + � µ , b � + ǫ || µ || Q � �� � � �� � ”Likelihood” ”Prior” Original problem. Convert constraints to a convex function. Use any norm... and relax constraints. Generalize SBG entropy to Bregman divergence. Find the Fenchel dual problem to solve. It’s a more general form of the MAP problem. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 7 / 36

  14. Generalizing Maxent Characterizing the solution Compare to statistical models After solving for ¯ µ we can recover the optimal primal solution: ”Score” ���� A T ¯ p = ∇ F ∗ µ + p ∗ ¯ ( 0 ) ���� ”Family” T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 8 / 36

  15. Generalizing Maxent Characterizing the solution Compare to statistical models After solving for ¯ µ we can recover the optimal primal solution: ”Score” ���� A T ¯ p = ∇ F ∗ µ + p ∗ ¯ ( 0 ) ���� ”Family” p comes from a family of distributions. ¯ T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 8 / 36

  16. Generalizing Maxent Characterizing the solution Compare to statistical models After solving for ¯ µ we can recover the optimal primal solution: ”Score” ���� A T ¯ p = ∇ F ∗ µ + p ∗ ¯ ( 0 ) ���� ”Family” p comes from a family of distributions. ¯ Entropy function ( F ) determines the family ( ∇ F ∗ ). T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 8 / 36

  17. Generalizing Maxent Characterizing the solution Compare to statistical models After solving for ¯ µ we can recover the optimal primal solution: ”Score” ���� A T ¯ p = ∇ F ∗ µ + p ∗ ¯ ( 0 ) ���� ”Family” p comes from a family of distributions. ¯ Entropy function ( F ) determines the family ( ∇ F ∗ ). SBG entropy → exponential family. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 8 / 36

  18. Generalizing Maxent Characterizing the solution Compare to statistical models After solving for ¯ µ we can recover the optimal primal solution: ”Score” ���� A T ¯ p = ∇ F ∗ µ + p ∗ ¯ ( 0 ) ���� ”Family” p comes from a family of distributions. ¯ Entropy function ( F ) determines the family ( ∇ F ∗ ). SBG entropy → exponential family. Any “nice” F → some family. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 8 / 36

  19. Generalizing Maxent Generalizing the Exponential Family q-Exponential exp q 8 q � 1.5 6 q � 1. 4 q � 0.5 2 Asymptote for q � 1.5 � 3 � 2 � 1 0 1 2 3  1  1 − q (1 + (1 − q ) p ) q � = 1 + exp q ( p ) :=  exp( p ) q = 1 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 9 / 36

  20. Generalizing Maxent Tail Behavior Tail Behavior exp q 1.0 0.8 0.6 0.4 0.2 � 3.0 � 2.5 � 2.0 � 1.5 � 1.0 � 0.5 0.0 q > 1 naturally gives fat tails. q < 1 truncates the tail. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 10 / 36

  21. Two Examples You are here Generalizing Maxent 1 Two Examples 2 Broader Comparisons 3 Extensions/Conclusions 4 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 11 / 36

  22. Two Examples Loaded Die Example Setup A die with 6 faces. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  23. Two Examples Loaded Die Example Setup A die with 6 faces. Expected value of 4.5, instead of 3.5 for a “fair die”. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  24. Two Examples Loaded Die Example Setup A die with 6 faces. Expected value of 4.5, instead of 3.5 for a “fair die”. For this problem: � 1 � 4 . 5 � � 2 3 4 5 6 A = and b = 1 1 1 1 1 1 1 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  25. Two Examples Loaded Die Example Setup A die with 6 faces. Expected value of 4.5, instead of 3.5 for a “fair die”. For this problem: � 1 � 4 . 5 � � 2 3 4 5 6 A = and b = 1 1 1 1 1 1 1 Find p , assuming S → S q , p 0 is uniform, ǫ = 0. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  26. Two Examples Loaded Die Example Setup A die with 6 faces. Expected value of 4.5, instead of 3.5 for a “fair die”. For this problem: � 1 � 4 . 5 � � 2 3 4 5 6 A = and b = 1 1 1 1 1 1 1 Find p , assuming S → S q , p 0 is uniform, ǫ = 0. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  27. Two Examples Loaded Die Example Setup A die with 6 faces. Expected value of 4.5, instead of 3.5 for a “fair die”. For this problem: � 1 � 4 . 5 � � 2 3 4 5 6 A = and b = 1 1 1 1 1 1 1 Find p , assuming S → S q , p 0 is uniform, ǫ = 0. T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 12 / 36

  28. Two Examples Loaded Die Example Probability q � Sensitivity of Each Event Varies q 0.3 0.1 1. 1.9 0.2 0.1 0.0 Higher q raises weight on face 1 and face 6. Opposite for 3,4,5. Task: Make a two-way market on each die face. Which is easiest? T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 13 / 36

  29. Two Examples Example: The Dantzig Selector Entropy Function as Prior Information Background: Consider a variation on linear regression � y = X β . Choose β via β || β || 1 + δ ǫ B ∞ ( X T ( X β − y )) min T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 14 / 36

  30. Two Examples Example: The Dantzig Selector Entropy Function as Prior Information Background: Consider a variation on linear regression � y = X β . Choose β via β || β || 1 + δ ǫ B ∞ ( X T ( X β − y )) min The non-zero entries of the solution can exactly identify the correct set of regressors with high probability under special conditions. (Candace and Tao, Ann. Stat. 2007) T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 14 / 36

Recommend

Overview MAXENT-Modeling: A framework for Discrete MAXENT-Models and RMs IRT-Modeling?

Overview MAXENT-Modeling: A framework for Discrete MAXENT-Models and RMs IRT-Modeling?

Overview MAXENT-Modeling: A framework for Discrete MAXENT-Models and RMs IRT-Modeling? Conceptual foundation and relationships of the MAXENT-framework The canonical MAXENT-distribution is structurally similar to Dipl.-Psych. Georg

366 views • 7 slides
From Python to PySpark and Back Again - Unifying Single-host and Distributed Machine Learning

From Python to PySpark and Back Again - Unifying Single-host and Distributed Machine Learning

From Python to PySpark and Back Again - Unifying Single-host and Distributed Machine Learning with Maggy Moritz Meister, @morimeister Software Engineer, Logical Clocks Jim Dowling, @jim_dowling Associate Professor, KTH Royal Institute of

505 views • 33 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Training Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Training Neural Nets 1 / 40 Outline 1 The Softmax Simplex 2 Loss and Risk 3 Back-Propagation 4 Stochastic Gradient Descent 5 Regularization 6 Network Depth

675 views • 40 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine Learning? What is Machine Learning? Examples of machine learning applications: Facial recognition software E-mail spam detectors Automated

302 views • 17 slides
Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning 11.1 Learning agents 11.2 Inductive learning 11.3 Deep learning 11.4 Statistical learning 11.5 Reinforcement learning 11.6 Transfer learning

1.7k views • 159 slides
Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine learning pipeline Temporal Frame-level Unsupervised Classifier Summary

613 views • 27 slides
Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine learning refers to any system where the performance of a machine in performing a task improves by gaining more experience in performing that task .

394 views • 27 slides
MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How

MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How

MACHINE LEARNING 2012 MACHINE LEARNING MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How to separate the red class from the grey class? x 2 360 r x 1 Polar coordinates Data

1.04k views • 44 slides
BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

photo:@rewardyfahmi // Unsplash BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest Neighbor Classifier Aykut Erdem // Hacettepe University // Fall 2019 When Do We Use Machine Learning? ML is

1.33k views • 110 slides
Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a

Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a

Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a dataset, identify features which may help predict information about the data Identify common types of machine learning algorithms 2 Machine

553 views • 27 slides
Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in

Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in

Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in Practise Probabilities Finite Resource Machine Learning @ Amazon Forecasting Machine Translation Visual Systems Conclusions

348 views • 34 slides
Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

CS6501: Deep Learning for Visual Recognition Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning? Supervised Learning: Classification with K-nearest neighbors Unsupervised Learning: Clustering with

1.36k views • 91 slides
Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton with Nitish Srivastava Kevin Swersky What is Machine Learning? It is very hard to write programs that solve problems like recognizing a

514 views • 48 slides
MACHINE LEARNING Overview 1 MACHINE LEARNING Oral Presentations of Projects Start at 9h15 am

MACHINE LEARNING Overview 1 MACHINE LEARNING Oral Presentations of Projects Start at 9h15 am

MACHINE LEARNING MACHINE LEARNING Overview 1 MACHINE LEARNING Oral Presentations of Projects Start at 9h15 am and last until 12h30 2 MACHINE LEARNING Exam Format The exam lasts a total of 40 minutes: -Upon entering the room, you pick at

919 views • 57 slides
Machine Learning for Music: Intro Juhan Nam Definition of Machine Learning Tom M. Mitchell

Machine Learning for Music: Intro Juhan Nam Definition of Machine Learning Tom M. Mitchell

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Machine Learning for Music: Intro Juhan Nam Definition of Machine Learning Tom M. Mitchell provided a widely accepted definition: A computer program is said to learn from

595 views • 24 slides

More recommend