cs480 680 machine learning lecture 3 may 13 2019
play

CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear - PowerPoint PPT Presentation

Oct 05, 2022 •428 likes •582 views

CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear Regression [RN] Sec. 18.6.1, [HTF] Sec. 2.3.1, [D] Sec. 7.6, [B] Sec. 3.1, [M] Sec. 1.4.5 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Linear model for regression

  1. CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear Regression [RN] Sec. 18.6.1, [HTF] Sec. 2.3.1, [D] Sec. 7.6, [B] Sec. 3.1, [M] Sec. 1.4.5 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1

  2. Linear model for regression • Simple form of regression • Picture: University of Waterloo CS480/680 Spring 2019 Pascal Poupart 2

  3. Problem • Data: { ! " , $ % , ! & , $ ' , … , (! * , $ + )} – ! = < 0 % , 0 ' , … , 0 1 > : input vector – $ : target (continuous value) • Problem: find hypothesis ℎ that maps ! to $ – Assume that ℎ is linear: 4 !, 5 = 6 7 + 6 % 0 % + ⋯ + 6 1 0 1 = 5 : 1 ! • Objective: minimize some loss function – Euclidean loss: < ' (5) = % + − $ > ' ' ∑ >?% 4 ! @ , 5 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 3

  4. Optimization • Find best ! that minimizes Euclidean loss 1 7 1 1 " ∗ = %&'()* " 2 . − " 4 2 - 5 6 ./0 • Convex optimization problem ⟹ unique optimum (global) University of Waterloo CS480/680 Spring 2019 Pascal Poupart 4

  5. ̅ Solution 1 - , 1 2 / − + 4 ! - ∑ /0, • Let ! " = " then min " 5 + • Find + ∗ by setting the derivative to 0 78 9 1 2 / − + 4 ! = ∑ /0, " 5 = /> = 0 ∀A 7 :; 1 2 / − + 4 ! ⟹ ∑ /0, " 5 ! " 5 = 0 • This is a linear system in + , therefore we rewrite it as C+ = D 4 and D = ∑ /0, 1 1 where C = ∑ /0, ! " 5 ! 2 / ! " 5 " 5 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 5

  6. Solution • If training instances span ℜ "#$ then % is invertible: & = % () * • In practice it is faster to solve the linear system %& = * directly instead of inverting % – Gaussian elimination – Conjugate gradient – Iterative methods University of Waterloo CS480/680 Spring 2019 Pascal Poupart 6

  7. Picture University of Waterloo CS480/680 Spring 2019 Pascal Poupart 7

  8. Regularization • Least square solution may not be stable – i.e., slight perturbation of the input may cause a dramatic change in the output – Form of overfitting University of Waterloo CS480/680 Spring 2019 Pascal Poupart 8

  9. Example 1 " # = 1 " ' = 1 • Training data: ! ! 0 ( ) * = 1 ) + = 1 • , = • , -# = . = • / = University of Waterloo CS480/680 Spring 2019 Pascal Poupart 9

  10. Example 2 " # = 1 " ' = 1 • Training data: ! ! 0 ( ) * = 1 + ( ) , = 1 • - = • - .# = / = • 0 = University of Waterloo CS480/680 Spring 2019 Pascal Poupart 10

  11. Picture University of Waterloo CS480/680 Spring 2019 Pascal Poupart 11

  12. Regularization • Idea: favor smaller values " as a penalty term • Tikhonov regularization: add ! " • Ridge regression: 1 1 " + 9 " ! ∗ = %&'()* ! 2 . − ! 4 5 2 - 6 7 2 ! " ./0 where 9 is a weight to adjust the importance of the penalty University of Waterloo CS480/680 Spring 2019 Pascal Poupart 12

  13. Regularization • Solution: !" + $ % = ' • Notes – Without regularization: eigenvalues of linear system may be arbitrarily close to 0 and the inverse may have arbitrarily large eigenvalues. – With Tikhonov regularization, eigenvalues of linear system are ≥ ! and therefore bounded away from 0. Similarly, eigenvalues of inverse are bounded above by 1/! . University of Waterloo CS480/680 Spring 2019 Pascal Poupart 13

  14. Regularized Examples Example 1 Example 2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 14

Recommend

CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart

CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart

CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Introduction to Machine Learning Course website and logistics University of

527 views • 24 slides
CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2

CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2

CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Statistical Learning View: we have uncertain knowledge of the world Idea:

1.43k views • 25 slides
CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D]

CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D]

CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D] Chapt. 3, [B] Sec. 2.5.2, [M] Sec. 1.4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Inductive Learning (recap) Induction

1.2k views • 25 slides
CS480/680 Machine Learning Lecture 6: January 23 st , 2020 Maximum A posteriori &amp; Maximum

CS480/680 Machine Learning Lecture 6: January 23 st , 2020 Maximum A posteriori &amp; Maximum

CS480/680 Machine Learning Lecture 6: January 23 st , 2020 Maximum A posteriori &amp; Maximum Likelihood Zahra Sheikhbahaee Sources: A Tutorial on Energy Based Learning University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee Outline

702 views • 17 slides
CS480/680 Machine Learning Lecture 1: January 7 th , 2020 Course Introduction Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 1: January 7 th , 2020 Course Introduction Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 1: January 7 th , 2020 Course Introduction Zahra Sheikhbahaee CS480/680 Winter 2020 Zahra Sheikhbahaee University of Waterloo 1 Outline Introduction to Machine Learning Course website and details:

404 views • 26 slides
CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra

CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra

CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra Sheikhbahaee University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee 1 Outline -mean Clustering Gaussian Mixture model EM for

535 views • 30 slides
CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29,

CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29,

CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29, 2020 University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee 1 Outline Convolution Zero Padding Stride Weight Sharing Pooling

1.24k views • 22 slides
CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B]

CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B]

CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B] Chap. 14, [HTF] Chap 15-16, [D] Chap. 11 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Ensemble Learning Bagging

1.72k views • 22 slides
CS480/680 Machine Learning Lecture 8: January 30 th , 2020 Graphical Models Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 8: January 30 th , 2020 Graphical Models Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 8: January 30 th , 2020 Graphical Models Zahra Sheikhbahaee University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee 1 Outline Graphical Model Bayesian Network Conditional Independency

885 views • 21 slides
CS480/680 Machine Learning Lecture 5: January 21 st , 2020 Information Theory Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 5: January 21 st , 2020 Information Theory Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 5: January 21 st , 2020 Information Theory Zahra Sheikhbahaee Sources: Elements Of Information Theory Information Theory, Inference, and Learning Algorithms University of Waterloo CS480/680 Winter 2020 Zahra

339 views • 16 slides
CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra Sheikhbahaee VARIATIONAL ALGORITHMS FOR APPROXIMATE BAYESIAN INFERENCE (Beal 2003, chapter 2) Variational Inference: A Review for Statisticians (Blei et al.

713 views • 28 slides
CS480/680 Machine Learning Lecture 3: January 14 th , 2020 Linear Regression Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 3: January 14 th , 2020 Linear Regression Zahra Sheikhbahaee

CS480/680 Machine Learning Lecture 3: January 14 th , 2020 Linear Regression Zahra Sheikhbahaee CS480/680 Winter 2020 Zahra Sheikhbahaee First Assignment Available 15 th of January Deadline 23 th of January Teacher assistants

664 views • 18 slides
CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2,

CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2,

CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2, [M] Section 4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Linear Models Probabilistic Generative Models Regression

557 views • 17 slides
CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear models [RN] Sec 18.6.4, [B] Sec. 4.3, [M] Chapt. 8, [HTF] Sec. 4.4 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Beyond Mixtures of

756 views • 18 slides
CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Variable length data Traditional feed forward neural networks can only handle fixed

609 views • 18 slides
CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec.

CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec.

CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec. 11.5-11.6 [HTF] Chap. 12 [M] Sec. 14.5 [RN] 18.9 [MRT] Chap. 4 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Overlapping Class

496 views • 14 slides
CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec.

CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec.

CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec. 8.3 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Gaussian Process Regression Idea: distribution over functions University of

526 views • 23 slides
CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al.,

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al.,

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al., Attention is All You Need, NeurIPS , 2017] University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Attention Attention in Computer Vision

708 views • 13 slides
CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec.

CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec.

CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec. 14.1, 14.2 [HTF] Chap. 6 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Non-linear Models Recap Generalized linear models:

611 views • 15 slides
CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Deep Neural Networks Gradient Vanishing Rectified linear units Overfitting

941 views • 25 slides
BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

photo:@rewardyfahmi // Unsplash BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest Neighbor Classifier Aykut Erdem // Hacettepe University // Fall 2019 When Do We Use Machine Learning? ML is

1.33k views • 110 slides
CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11,

CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11,

CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11, [B] Sec. 4.1.7, 5.1, [M] Sec. 8.5.4, [RN] Sec. 18.7 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Neural networks

885 views • 29 slides
CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10,

CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10,

CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10, [M] Sec. 16.2.5, 16.4.5 [B] Chap. 14, [HTF] Chap 10, 15-16, [D] Chap. 13 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Gradient

1.45k views • 20 slides
Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton with Nitish Srivastava Kevin Swersky What is Machine Learning? It is very hard to write programs that solve problems like recognizing a

514 views • 48 slides

More recommend