lecture 3 linear classification
play

Lecture 3: Linear Classification Fei-Fei Li & Andrej Karpathy - PowerPoint PPT Presentation

Jan 01, 2023 •719 likes •1.2k views

Lecture 3: Linear Classification Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 1 Last time: Image Classification assume given set of discrete labels {dog, cat, truck,

  1. Lecture 3: Linear Classification Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 1

  2. Last time: Image Classification assume given set of discrete labels {dog, cat, truck, plane, ...} cat Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 2

  3. k-Nearest Neighbor test images training set Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 3

  4. Linear Classification 1. define a score function class scores Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 4

  5. Linear Classification 1. define a score function data (image) “bias vector” “weights” class scores “parameters” Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 5

  6. Linear Classification data (image) 1. define a score function [3072 x 1] (assume CIFAR-10 example so 32 x 32 x 3 images, 10 classes) bias vector weights class scores Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 6

  7. Linear Classification data (image) 1. define a score function [3072 x 1] (assume CIFAR-10 example so 32 x 32 x 3 images, 10 classes) bias vector weights class scores [10 x 1] [10 x 3072] [10 x 1] Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 7

  8. Linear Classification Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 8

  9. Interpreting a Linear Classifier Question: what can a linear classifier do? Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 9

  10. Interpreting a Linear Classifier Example training classifiers on CIFAR-10: Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 10

  11. Interpreting a Linear Classifier Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 11

  12. Bias trick Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 12

  13. So far: We defined a (linear) score function: Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 13

  14. 2. Define a loss function (or cost function, or objective) Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 14

  15. 2. Define a loss function (or cost function, or objective) - scores, label loss . Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 15

  16. 2. Define a loss function (or cost function, or objective) - scores, label loss . Example: Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 16

  17. 2. Define a loss function (or cost function, or objective) - scores, label loss . Question: if you were to Example: assign a single number to how “unhappy” you are with these scores, what would you do? Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 17

  18. 2. Define a loss function (or cost function, or objective) One (of many ways) to do it: Multiclass SVM Loss Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 18

  19. 2. Define a loss function (or cost function, or objective) One (of many ways) to do it: Multiclass SVM Loss (One possible generalization of Binary Support Vector Machine to multiple classes) Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 19

  20. 2. Define a loss function (or cost function, or objective) One (of many ways) to do it: Multiclass SVM Loss loss due to difference between the correct class sum over all example i score and incorrect class score incorrect labels Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 20

  21. loss due to difference between the correct class sum over all example i score and incorrect class score incorrect labels Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 21

  22. Example: e.g. 10 loss = ? Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 22

  23. Example: e.g. 10 Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 23

  24. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 24

  25. There is a bug with the objective… Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 25

  26. L2 Regularization Regularization strength Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 26

  27. L2 regularization: motivation Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 27

  28. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 28

  29. Do we have to cross-validate both and ? Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 29

  30. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 30

  31. So far… 1. Score function 2. Loss function Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 31

  32. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 32

  33. score function Softmax Classifier is the same (extension of Logistic Regression to multiple classes) Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 33

  34. score function Softmax Classifier is the same Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 34

  35. score function Softmax Classifier is the same softmax function i.e. we’re minimizing the negative log likelihood. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 35

  36. score function Softmax Classifier is the same softmax function Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 36

  37. score function Softmax Classifier is the same i.e. we’re minimizing the negative log likelihood. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 37

  38. Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 38

  39. Softmax vs. SVM - Interpreting the probabilities from the Softmax Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 39

  40. Softmax vs. SVM - Interpreting the probabilities from the Softmax suppose the weights W were only half as large (we use a higher regularization strength) Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 40

  41. Softmax vs. SVM - Interpreting the probabilities from the Softmax suppose the weights W were only half as large: Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 41

  42. Softmax vs. SVM - Interpreting the probabilities from the Softmax suppose the weights W were only half as large: What happens in the limit, as the regularization strength goes to infinity? Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 42

  43. Softmax vs. SVM 1 scores: [10, -2, 3] [10, 9, 9] [10, -100, -100] Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 43

  44. Softmax vs. SVM 1 Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 44

  45. Interactive Web Demo time.... http://vision.stanford.edu/teaching/cs231n/linear-classify-demo/ Fei-Fei Li & Andrej Karpathy Fei-Fei Li & Andrej Karpathy Lecture 2 - Lecture 2 - 7 Jan 2015 7 Jan 2015 45

Recommend

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear

Learning From Data Lecture 8 Linear Classification and Regression Linear Classification Linear Regression M. Magdon-Ismail CSCI 4100/6100 recap: Approximation Versus Generalization VC Analysis Bias-Variance Analysis E out E in + ( d

390 views • 23 slides
Homework Homework Lecture 7: Linear Classification Methods Final projects? Groups Topics

Homework Homework Lecture 7: Linear Classification Methods Final projects? Groups Topics

Homework Homework Lecture 7: Linear Classification Methods Final projects? Groups Topics Proposal week 5 Lecture 20 is poster session, Jacobs Hall Lobby, snacks Final report 15 June. What is linear classification? Classification is

524 views • 26 slides
Web Information Retrieval Lecture 14 Text classification Sec. 13.1 Text Classification

Web Information Retrieval Lecture 14 Text classification Sec. 13.1 Text Classification

Web Information Retrieval Lecture 14 Text classification Sec. 13.1 Text Classification Nave Bayes Classification Vector space methods for Text Classification K Nearest Neighbors Decision boundaries Linear Classifiers

744 views • 47 slides
CSC 411 Lecture 8: Linear Classification II Roger Grosse, Amir-massoud Farahmand, and Juan

CSC 411 Lecture 8: Linear Classification II Roger Grosse, Amir-massoud Farahmand, and Juan

CSC 411 Lecture 8: Linear Classification II Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 08-Linear Classification 1 / 34 Todays Agenda Todays agenda: Gradient checking with finite

597 views • 34 slides
Data Mining Lecture 03: Introduction to classification Linear classifier Theses

Data Mining Lecture 03: Introduction to classification Linear classifier Theses

CISC 4631 Data Mining Lecture 03: Introduction to classification Linear classifier Theses slides are based on the slides by Tan, Steinbach and Kumar (textbook authors) Eamonn Koegh (UC Riverside) 1 Classification:

650 views • 32 slides
Lecture 9: Logistic Regression Discriminative vs. Generative Classification Linear

Lecture 9: Logistic Regression Discriminative vs. Generative Classification Linear

Lecture 9: Logistic Regression Discriminative vs. Generative Classification Linear Discriminant Functions Aykut Erdem October 2017 Hacettepe University Administrative Midterm exam will be held on November 6. Project proposals

562 views • 52 slides
Extending Binary Linear Classification One-Versus-All Classification (OVA) } In the presence of

Extending Binary Linear Classification One-Versus-All Classification (OVA) } In the presence of

Binary and Other Classification } We will generally discuss binary classifiers, which divide data into one of two classes } Linear classifiers as presented in last lecture are inherently binary, defining the classes based on two regions,

336 views • 7 slides
Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335

Logistic Regression Lecture 6 Logistic Regression Classification Model CS 335 Cost function Gradient descent Linear classifiers and decision boundaries Dan Sheldon Classification Example: Hand-Written Digits Input: 20

541 views • 5 slides
Linear Regression 1 / 10 The Linear Model So far weve dealt with classification, where the

Linear Regression 1 / 10 The Linear Model So far weve dealt with classification, where the

Linear Regression 1 / 10 The Linear Model So far weve dealt with classification, where the target function maps feature vectors to discrete classes, but the linear model is more versatile. Consider the credit analysis problem: We can use

566 views • 10 slides
Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear

Multiclass Classification Machine Learning So far: Binary Classification We have seen linear models Learning algorithms for linear models Perceptron, Winnow, Adaboost, SVM We will see more soon: Nave Bayes, Logistic Regression

635 views • 24 slides
Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC

Lecture #10: Classification & Logistic Regression Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Module 2: Classification Why not Linear Regression? Binary

1.1k views • 77 slides
CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear models [RN] Sec 18.6.4, [B] Sec. 4.3, [M] Chapt. 8, [HTF] Sec. 4.4 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Beyond Mixtures of

756 views • 18 slides
Algorithms for NLP CS 11-711 Fall 2020 Lecture 2: Linear text classification Emma Strubell

Algorithms for NLP CS 11-711 Fall 2020 Lecture 2: Linear text classification Emma Strubell

Algorithms for NLP CS 11-711 Fall 2020 Lecture 2: Linear text classification Emma Strubell Lets try this again Emma Yulia Bob Sanket Han Jiateng she/her she/her he/him he/him he/him he/him 2 Outline Basic

534 views • 37 slides
E9 205 Machine Learning for Signal Processing Probablistic Linear Models 30-09-2019 Linear

E9 205 Machine Learning for Signal Processing Probablistic Linear Models 30-09-2019 Linear

E9 205 Machine Learning for Signal Processing Probablistic Linear Models 30-09-2019 Linear Models for Classification Optimize a modified cost function Bishop - PRML book (Chap 3) Least Squares for Classification K-class classification

423 views • 6 slides
Linear Classification 1 / 14 The Linear Model In the next few lectures we will extend the

Linear Classification 1 / 14 The Linear Model In the next few lectures we will extend the

Linear Classification 1 / 14 The Linear Model In the next few lectures we will extend the perceptron learning algorithm to handle non-linearly separable data, explore online versis batch learning, learn three different learning

466 views • 14 slides
CSE 258 Lecture 3 Web Mining and Recommender Systems Supervised learning Classification

CSE 258 Lecture 3 Web Mining and Recommender Systems Supervised learning Classification

CSE 258 Lecture 3 Web Mining and Recommender Systems Supervised learning Classification Last week Last week we started looking at supervised learning problems Last week We studied linear regression , in order to learn linear

987 views • 67 slides
Linear Coding of non-linear Hierarchies: Revitalization of an Ancient Classification Method

Linear Coding of non-linear Hierarchies: Revitalization of an Ancient Classification Method

Problem P an .inis Solution Generalization Transfer Linear Coding of non-linear Hierarchies: Revitalization of an Ancient Classification Method Wiebke Petersen Institute of Language and Information University of Dsseldorf

701 views • 47 slides
Classification James H. Steiger Department of Psychology and Human Development Vanderbilt

Classification James H. Steiger Department of Psychology and Human Development Vanderbilt

Introduction The Linear Classification Function Quadratic Classification Functions Estimating Misclassification Rates Bias in Error Rate Estimation Error Rates in Variable Selection Classification via the k Nearest Neighbor Rule Classification

417 views • 31 slides
Data Classification Linear Classifier II Latent Differential Analysis Mean Classification

Data Classification Linear Classifier II Latent Differential Analysis Mean Classification

Data Classification Linear Classifier II Latent Differential Analysis Mean Classification Memory If youre here, you are RED If youre here, you are BLUE 2 Back Linear Classifier A classifier that assigns a class to a new point

637 views • 36 slides
Natural Language Processing Classification Classification II Dan Klein UC Berkeley Linear

Natural Language Processing Classification Classification II Dan Klein UC Berkeley Linear

Natural Language Processing Classification Classification II Dan Klein UC Berkeley Linear Models: Perceptron Issues with Perceptrons The perceptron algorithm Overtraining: test / held out accuracy Iteratively processes the

259 views • 12 slides
Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review:

Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review:

Machine Learning Basics Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review: machine learning basics Math formulation Given training data , : 1 i.i.d. from

2.11k views • 34 slides
Linear Classification Linear separability Inseparability Real world problems: there may not

Linear Classification Linear separability Inseparability Real world problems: there may not

Linear Classification Linear separability Inseparability Real world problems: there may not exist a hyperplane that separates cleanly Solution to this inseparability problem: map data to higher dimensional space Called the

461 views • 27 slides
Classification K-nearest neighbor classification D istance functions Choice of k Choice of k

Classification K-nearest neighbor classification D istance functions Choice of k Choice of k

Classification K-nearest neighbor classification D istance functions Choice of k Choice of k Leave-one-out cross validation K-fold cross validation Classification Error = Average classification error on K folds Linear Classification Linear

395 views • 22 slides
Non-linear hypotheses Machine Learning Non-linear Classification x 2 x 1 size # bedrooms #

Non-linear hypotheses Machine Learning Non-linear Classification x 2 x 1 size # bedrooms #

Neural Networks: Representation Non-linear hypotheses Machine Learning Non-linear Classification x 2 x 1 size # bedrooms # floors age Andrew Ng What is this? You see this: But the camera sees this: Andrew Ng Computer Vision: Car

470 views • 42 slides

More recommend