lecture 3 linear classifiers
play

Lecture 3: Linear Classifiers Justin Johnson Lecture 3 - 1 - PowerPoint PPT Presentation

Sep 10, 2022 •231 likes •1.19k views

Lecture 3: Linear Classifiers Justin Johnson Lecture 3 - 1 September 11, 2019 Reminder: Assignment 1 http://web.eecs.umich.edu/~justincj/teaching/eecs498/assignment1.html Due Sunday September 15, 11:59pm EST We have written a

  1. Lecture 3: Linear Classifiers Justin Johnson Lecture 3 - 1 September 11, 2019

  2. Reminder: Assignment 1 • http://web.eecs.umich.edu/~justincj/teaching/eecs498/assignment1.html • Due Sunday September 15, 11:59pm EST • We have written a homework validation script to check the format of your .zip file before you submit to Canvas: • https://github.com/deepvision-class/tools#homework- validation • This script ensures that your .zip and .ipynb files are properly structured; they do not check correctness • It is your responsibility to make sure your submitted .zip file passes the validation script Justin Johnson Lecture 3 - 2 September 11, 2019

  3. Last time: Image Classification Input : image Output : Assign image to one of a fixed set of categories cat bird deer dog truck This image by Nikita is licensed under CC-BY 2.0 Justin Johnson Lecture 3 - 3 September 11, 2019

  4. Last Time: Challenges of Recognition Illumination Deformation Occlusion Viewpoint This image by Umberto Salvagnin is This image is CC0 1.0 public domain This image by jonsson is licensed licensed under CC-BY 2.0 under CC-BY 2.0 Clutter Intraclass Variation This image is CC0 1.0 public domain This image is CC0 1.0 public domain Justin Johnson Lecture 3 - 4 September 11, 2019

  5. Last time: Data-Drive Approach, kNN 1-NN classifier 5-NN classifier train test train validation test Justin Johnson Lecture 3 - 5 September 11, 2019

  6. Today: Linear Classifiers Justin Johnson Lecture 3 - 6 September 11, 2019

  7. Neural Network Linear classifiers This image is CC0 1.0 public domain Justin Johnson Lecture 3 - 7 September 11, 2019

  8. Recall CIFAR10 50,000 training images each image is 32x32x3 10,000 test images. Justin Johnson Lecture 3 - 8 September 11, 2019

  9. Parametric Approach Image 10 numbers giving f( x , W ) class scores Array of 32x32x3 numbers W (3072 numbers total) parameters or weights Justin Johnson Lecture 3 - 9 September 11, 2019

  10. Parametric Approach: Linear Classifier f(x,W) = Wx Image 10 numbers giving f( x , W ) class scores Array of 32x32x3 numbers W (3072 numbers total) parameters or weights Justin Johnson Lecture 3 - 10 September 11, 2019

  11. Parametric Approach: Linear Classifier (3072,) f(x,W) = Wx Image (10,) (10, 3072) 10 numbers giving f( x , W ) class scores Array of 32x32x3 numbers W (3072 numbers total) parameters or weights Justin Johnson Lecture 3 - 11 September 11, 2019

  12. Parametric Approach: Linear Classifier (3072,) f(x,W) = Wx + b (10,) Image (10,) (10, 3072) 10 numbers giving f( x , W ) class scores Array of 32x32x3 numbers W (3072 numbers total) parameters or weights Justin Johnson Lecture 3 - 12 September 11, 2019

  13. Example for 2x2 image, 3 classes (cat/dog/ship) Stretch pixels into column f(x,W) = Wx + b 56 56 231 231 24 2 24 Input image 2 (2, 2) (4,) Justin Johnson Lecture 3 - 13 September 11, 2019

  14. Example for 2x2 image, 3 classes (cat/dog/ship) Stretch pixels into column f(x,W) = Wx + b 56 0.2 -0.5 0.1 2.0 1.1 -96.8 56 231 231 + = 1.5 1.3 2.1 0.0 3.2 437.9 24 2 24 0 0.25 0.2 -0.3 -1.2 61.95 Input image 2 (2, 2) W b (3,) (3, 4) (4,) (3,) Justin Johnson Lecture 3 - 14 September 11, 2019

  15. Linear Classifier: Algebraic Viewpoint Stretch pixels into column f(x,W) = Wx + b 56 0.2 -0.5 0.1 2.0 1.1 -96.8 56 231 231 + = 1.5 1.3 2.1 0.0 3.2 437.9 24 2 24 0 0.25 0.2 -0.3 -1.2 61.95 Input image 2 (2, 2) W b (3,) (3, 4) (4,) (3,) Justin Johnson Lecture 3 - 15 September 11, 2019

  16. Add extra one to data vector; Linear Classifier: Bias Trick bias is absorbed into last column of weight matrix Stretch pixels into column 56 0.2 -0.5 0.1 2.0 1.1 -96.8 56 231 231 = 1.5 1.3 2.1 0.0 3.2 437.9 24 2 24 0 0.25 0.2 -0.3 -1.2 61.95 Input image 2 (2, 2) W (3,) (3, 5) (5,) 1 Justin Johnson Lecture 3 - 16 September 11, 2019

  17. Linear Classifier: Predictions are Linear! f(x, W) = Wx (ignore bias) f(cx, W) = W(cx) = c * f(x, W) Justin Johnson Lecture 3 - 17 September 11, 2019

  18. Linear Classifier: Predictions are Linear! f(x, W) = Wx (ignore bias) f(cx, W) = W(cx) = c * f(x, W) Image Scores 0.5 * Scores 0.5 * Image -96.8 -48.4 218.9 437.8 62.0 31.0 Justin Johnson Lecture 3 - 18 September 11, 2019

  19. Interpreting a Linear Classifier Algebraic Viewpoint f(x,W) = Wx + b Stretch pixels into column 56 0.2 -0.5 0.1 2.0 1.1 -96.8 56 231 231 + = 1.5 1.3 2.1 0.0 3.2 437.9 24 2 24 0 0.25 0.2 -0.3 -1.2 61.95 Input image 2 (2, 2) W b (3,) (3, 4) (4,) (3,) Justin Johnson Lecture 3 - 19 September 11, 2019

  20. Interpreting a Linear Classifier Algebraic Viewpoint f(x,W) = Wx + b Stretch pixels into column 0.2 -0.5 1.5 1.3 0 .25 W 56 0.2 -0.5 0.1 2.0 1.1 -96.8 56 231 0.1 2.0 2.1 0.0 0.2 -0.3 231 + = 1.5 1.3 2.1 0.0 3.2 437.9 24 2 24 0 0.25 0.2 -0.3 -1.2 61.95 Input image 2 b (2, 2) W 1.1 3.2 b -1.2 (3,) (3, 4) (4,) (3,) -96.8 437.9 61.95 Justin Johnson Lecture 3 - 20 September 11, 2019

  21. Interpreting an Linear Classifier 0.2 -0.5 1.5 1.3 0 .25 W 0.1 2.0 2.1 0.0 0.2 -0.3 b 1.1 3.2 -1.2 -96.8 437.9 61.95 Justin Johnson Lecture 3 - 21 September 11, 2019

  22. Interpreting an Linear Classifier: Visual Viewpoint 0.2 -0.5 1.5 1.3 0 .25 W 0.1 2.0 2.1 0.0 0.2 -0.3 b 1.1 3.2 -1.2 -96.8 437.9 61.95 Justin Johnson Lecture 3 - 22 September 11, 2019

  23. Interpreting an Linear Classifier: Visual Viewpoint Linear classifier has one “template” per category 0.2 -0.5 1.5 1.3 0 .25 W 0.1 2.0 2.1 0.0 0.2 -0.3 b 1.1 3.2 -1.2 -96.8 437.9 61.95 Justin Johnson Lecture 3 - 23 September 11, 2019

  24. Interpreting an Linear Classifier: Visual Viewpoint Linear classifier has one “template” per category 0.2 -0.5 1.5 1.3 0 .25 W A single template cannot capture 0.1 2.0 2.1 0.0 0.2 -0.3 multiple modes of the data b 1.1 3.2 -1.2 e.g. horse template has 2 heads! -96.8 437.9 61.95 Justin Johnson Lecture 3 - 24 September 11, 2019

  25. Interpreting a Linear Classifier: Geometric Viewpoint f(x,W) = Wx + b Airplane Score Deer Score Classifier score Car Score Array of 32x32x3 numbers (3072 numbers total) Value of pixel (15, 8, 0) Justin Johnson Lecture 3 - 25 September 11, 2019

  26. Interpreting a Linear Classifier: Geometric Viewpoint Pixel f(x,W) = Wx + b (11, 11, 0) Car score increases this way Pixel (15, 8, 0) Array of 32x32x3 numbers Car Score (3072 numbers total) = 0 Justin Johnson Lecture 3 - 26 September 11, 2019

  27. Interpreting a Linear Classifier: Geometric Viewpoint Car template Pixel on this line f(x,W) = Wx + b (11, 11, 0) Car score increases this way Pixel (15, 8, 0) Array of 32x32x3 numbers Car Score (3072 numbers total) = 0 Justin Johnson Lecture 3 - 27 September 11, 2019

  28. Interpreting a Linear Classifier: Geometric Viewpoint Car template Airplane Pixel on this line f(x,W) = Wx + b Score (11, 11, 0) Car score increases this way Pixel (15, 8, 0) Array of 32x32x3 numbers Car Score (3072 numbers total) = 0 Deer Score Justin Johnson Lecture 3 - 28 September 11, 2019

  29. Interpreting a Linear Classifier: Geometric Viewpoint Car template Airplane Pixel Hyperplanes carving up a on this line Score (11, 11, 0) high-dimensional space Car score increases this way Pixel (15, 8, 0) Car Score = 0 Deer Score Plot created using Wolfram Cloud Justin Johnson Lecture 3 - 29 September 11, 2019

  30. Hard Cases for a Linear Classifier Class 1 : Class 1 : Class 1 : First and third quadrants 1 <= L2 norm <= 2 Three modes Class 2 : Class 2 : Class 2 : Everything else Everything else Second and fourth quadrants Justin Johnson Lecture 3 - 30 September 11, 2019

  31. Recall: Perceptron couldn’t learn XOR y X Y F(x,y) 0 0 0 0 1 1 1 0 1 1 1 0 x Justin Johnson Lecture 3 - 31 September 11, 2019

  32. Linear Classifier: Three Viewpoints Visual Viewpoint Algebraic Viewpoint Geometric Viewpoint One template Hyperplanes f(x,W) = Wx per class cutting up space Justin Johnson Lecture 3 - 32 September 11, 2019

  33. f(x,W) = Wx + b So Far: Defined a linear score funct ction Given a W, we can compute class scores for an image x. -3.45 -0.51 3.42 -8.87 6.04 4.64 0.09 5.31 2.65 But how can we 2.9 -4.22 5.1 actually choose a 4.48 -4.19 2.64 8.02 3.58 5.55 good W? 3.78 4.49 -4.34 1.06 -4.37 -1.5 -0.36 -2.09 -4.79 -0.72 -2.93 6.14 Cat image by Nikita is licensed under CC-BY 2.0; Car image is CC0 1.0 public domain; Frog image is in the public domain Justin Johnson Lecture 3 - 33 September 11, 2019

  34. f(x,W) = Wx + b Choosing a good W TODO: 1. Use a loss function to -3.45 -0.51 3.42 quantify how good a -8.87 6.04 4.64 value of W is 0.09 5.31 2.65 2.9 -4.22 5.1 4.48 -4.19 2. Find a W that minimizes 2.64 8.02 3.58 5.55 the loss function 3.78 4.49 -4.34 ( optimization ) 1.06 -4.37 -1.5 -0.36 -2.09 -4.79 -0.72 -2.93 6.14 Justin Johnson Lecture 3 - 34 September 11, 2019

  35. Loss Function A loss function tells how good our current classifier is Low loss = good classifier High loss = bad classifier (Also called: objective function ; cost function ) Justin Johnson Lecture 3 - 35 September 11, 2019

Recommend

CS440/ECE448 Lecture 22: Including Slides by Svetlana Lazebnik, 10/2016 Linear Classifiers

CS440/ECE448 Lecture 22: Including Slides by Svetlana Lazebnik, 10/2016 Linear Classifiers

Mark Hasegawa-Johnson, 3/2020 CS440/ECE448 Lecture 22: Including Slides by Svetlana Lazebnik, 10/2016 Linear Classifiers License: CC-BY 4.0 Linear Classifiers Classifiers Perceptron Linear classifiers in general Logistic

581 views • 45 slides
Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised

Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised

1 Nonlinear Classifiers II 2 Nonlinear Classifiers: Introduction Classifiers Supervised Classifiers XOR problem Linear Classifiers Perceptron Least Squares Methods Linear Support Vector Machine Nonlinear

355 views • 20 slides
Linear Classifiers: Expressiveness Machine Learning 1 Lecture outline Linear models:

Linear Classifiers: Expressiveness Machine Learning 1 Lecture outline Linear models:

Linear Classifiers: Expressiveness Machine Learning 1 Lecture outline Linear models: Introduction What functions do linear classifiers express? 2 Where are we? Linear models: Introduction What functions do linear classifiers

459 views • 27 slides
Linear Classifiers and the Perceptron William Cohen February 4, 2008 1 Linear classifiers

Linear Classifiers and the Perceptron William Cohen February 4, 2008 1 Linear classifiers

Linear Classifiers and the Perceptron William Cohen February 4, 2008 1 Linear classifiers Lets assume that every instance is an n -dimensional vector of real numbers x R n , and there are only two possible classes, y = (+1) and y = (

269 views • 9 slides
Lecture 14: Linear Classifiers Justin Johnson EECS 442 WI 2020: Lecture 14 - 1 February 25,

Lecture 14: Linear Classifiers Justin Johnson EECS 442 WI 2020: Lecture 14 - 1 February 25,

Lecture 14: Linear Classifiers Justin Johnson EECS 442 WI 2020: Lecture 14 - 1 February 25, 2020 Administrative HW3 due Wednesday, March 4 11:59pm TAs will not be checking Piazza over Spring Break. You are strongly encouraged to finish

869 views • 63 slides
CSC 311: Introduction to Machine Learning Lecture 3 - Linear Classifiers, Logistic Regression,

CSC 311: Introduction to Machine Learning Lecture 3 - Linear Classifiers, Logistic Regression,

CSC 311: Introduction to Machine Learning Lecture 3 - Linear Classifiers, Logistic Regression, Multiclass Classification Roger Grosse Chris Maddison Juhan Bae Silviu Pitis University of Toronto, Fall 2020 Intro ML (UofT) CSC311-Lec3 1 /

699 views • 48 slides
Extending Binary Linear Classification One-Versus-All Classification (OVA) } In the presence of

Extending Binary Linear Classification One-Versus-All Classification (OVA) } In the presence of

Binary and Other Classification } We will generally discuss binary classifiers, which divide data into one of two classes } Linear classifiers as presented in last lecture are inherently binary, defining the classes based on two regions,

336 views • 7 slides
Linear Classifiers and Perceptron E.D. And F. MAN TO BUY INTO

Linear Classifiers and Perceptron E.D. And F. MAN TO BUY INTO

Text Classification Linear Classifiers and Perceptron E.D. And F. MAN TO BUY INTO HONG KONG FIRM CS678 Advanced Topics in Machine Learning The U.K. Based commodity

157 views • 3 slides
Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2016 Topics Discriminant functions Linear classifiers Perceptron SVM will be covered in the later lectures Fisher Multi-class

616 views • 48 slides
Sketching Linear Classifiers Over Data Streams Kai Sheng Tai Vatsal Sharan, Peter Bailis,

Sketching Linear Classifiers Over Data Streams Kai Sheng Tai Vatsal Sharan, Peter Bailis,

Sketching Linear Classifiers Over Data Streams Kai Sheng Tai Vatsal Sharan, Peter Bailis, Gregory Valiant Stanford University High-dimensional linear classifiers on streams Ubiquitous: spam detection, ad click prediction, network traffic

604 views • 28 slides
Lecture #18: Support Vector Classifiers Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A

Lecture #18: Support Vector Classifiers Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A

Lecture #18: Support Vector Classifiers Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Rahul Dave Margo Levine Lecture Outline Classifying Linear Separable Data Classifying Linear Non-Separable Data 2

671 views • 22 slides
Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2019 Topics } Linear classifiers } Perceptron SVM will be covered in the later lectures } Fisher } Multi-class classification 2 Classification

881 views • 55 slides
Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall

Linear classifiers CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2018 Topics } Linear classifiers } Perceptron SVM will be covered in the later lectures } Fisher } Multi-class classification 2 Classification

786 views • 50 slides
Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied

Logistic regression CS 446 1. Linear classifiers Linear regression Last two lectures, we studied linear regression ; the output/label space Y was R . 90 80 delay 70 60 50 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 duration 1 / 68 Linear

607 views • 38 slides
Linear, Binary SVM Classifiers COMPSCI 371D Machine Learning COMPSCI 371D Machine

Linear, Binary SVM Classifiers COMPSCI 371D Machine Learning COMPSCI 371D Machine

Linear, Binary SVM Classifiers COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Linear, Binary SVM Classifiers 1 / 17 Outline 1 What Linear, Binary SVM Classifiers Do 2 Margin 3 Loss and Regularized Risk 4 Training an SVM is

510 views • 17 slides
Linear Classifiers and Regressors Borrowed with permission from Andrew Moore (CMU)

Linear Classifiers and Regressors Borrowed with permission from Andrew Moore (CMU)

Linear Classifiers and Regressors Borrowed with permission from Andrew Moore (CMU) Single-Parameter Linear Regression Linear: Slide 2 Regression vs Classification Input Prediction of Classifier Attributes categorical output Input

1.08k views • 48 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

839 views • 58 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

1.74k views • 60 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

850 views • 46 slides
CSCE 970 Lecture 4: Thus we will remap feature vectors to new Nonlinear Classifiers space

CSCE 970 Lecture 4: Thus we will remap feature vectors to new Nonlinear Classifiers space

Introduction For non-linearly separable classes, performance of even the best linear classifier might not be good CSCE 970 Lecture 4: Thus we will remap feature vectors to new Nonlinear Classifiers space where they are (almost) linearly

254 views • 11 slides
Outline CS 188: Artificial Intelligence Generative vs. Discriminative Binary Linear

Outline CS 188: Artificial Intelligence Generative vs. Discriminative Binary Linear

Outline CS 188: Artificial Intelligence Generative vs. Discriminative Binary Linear Classifiers Perceptron Lecture 21: Perceptrons Multi-class Linear Classifiers Multi-class Perceptron Fixing the Perceptron:

182 views • 7 slides
First look at structures CS 6355: Structured Prediction 1 So far Binary classifiers

First look at structures CS 6355: Structured Prediction 1 So far Binary classifiers

First look at structures CS 6355: Structured Prediction 1 So far Binary classifiers Output: 0/1 Multiclass classifiers Output: one of a set of labels Linear classifiers for both Learning algorithms Winner-take-all

1.05k views • 75 slides
CSE 258 Lecture 4 Web Mining and Recommender Systems Evaluating Classifiers Last lecture

CSE 258 Lecture 4 Web Mining and Recommender Systems Evaluating Classifiers Last lecture

CSE 258 Lecture 4 Web Mining and Recommender Systems Evaluating Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I

1.15k views • 46 slides
L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And their limitations: http://www.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node19.html What if we could use multiple classifiers?

579 views • 19 slides

More recommend