http cs246 stanford edu high dimension many features find
play

http://cs246.stanford.edu High-dimension == many features Find - PowerPoint PPT Presentation

Sep 28, 2022 •378 likes •928 views

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu High-dimension == many features Find concepts/topics/genres: Documents: Features: thousands of words, millions of word pairs Surveys

  1. CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu

  2.  High-dimension == many features  Find concepts/topics/genres:  Documents:  Features: thousands of words, millions of word pairs  Surveys – Netflix: 480k users x 177k movies 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 2

  3.  Compress / reduce dimensionality:  10 6 rows; 10 3 columns; no updates  random access to any cell(s); small error: OK 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 3

  4.  Assumption: Data lies on or near a low d -dimensional subspace  Axes of this subspace are effective representation of the data 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 4

  5.  Why reduce dimensionality?  Discover hidden correlations/topics  Words that occur commonly together  Remove redundant and noisy features  Not all words are useful  Interpretation and visualization  Easier storage and processing of the data 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 5

  6. A [n x m] = U [n x r] Σ [ r x r] ( V [m x r] ) T  A : Input data matrix  n x m matrix (e.g., n documents, m terms)  U : Left singular vectors  n x r matrix (n documents, r concepts)  Σ : Singular values  r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix)  V : Right singular vectos  m x r matrix (m terms, r concepts) 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 6

  7. n n ≈ Σ V T m m A U 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 7

  8. n σ 1 u 1 v 1 σ 2 u 2 v 2 ≈ + m A σ i … scalar u i … vector v i … vector 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 8

  9. It is always possible to decompose a real matrix A into A = U Σ V T , where  U, Σ , V : unique  U, V : column orthonormal:  U T U = I ; V T V = I ( I : identity matrix)  (Cols. are orthogonal unit vectors)  Σ : diagonal  Entries (singular values) are positive, and sorted in decreasing order ( σ 1 ≥ σ 2 ≥ σ 3 ≥ ...) 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 9

  10.  A = U Σ V T - example: Casablanca Serenity Amelie Matrix Alien 1 1 1 0 0 0.18 0 2 2 2 0 0 0.36 0 SciFi 1 1 1 0 0 9.64 0 0.18 0 x x = 5 5 5 0 0 0 5.29 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 10

  11.  A = U Σ V T - example: Casablanca SciFi-concept Serenity Amelie Matrix Romance-concept Alien 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 SciFi 9.64 0 0.18 0 1 1 1 0 0 x x = 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 11

  12.  A = U Σ V T - example: user-to-concept similarity matrix Casablanca SciFi-concept Serenity Amelie Matrix Romance-concept Alien 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 SciFi 9.64 0 0.18 0 1 1 1 0 0 x x = 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 12

  13.  A = U Σ V T - example: Casablanca Serenity Amelie Matrix Alien ‘strength’ of SciFi-concept 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 SciFi 9.64 0 0.18 0 1 1 1 0 0 x x = 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 13

  14.  A = U Σ V T - example: movie-to-concept Casablanca similarity matrix Serenity Amelie Matrix Alien 0.18 0 SciFi-concept 1 1 1 0 0 0.36 0 2 2 2 0 0 SciFi 9.64 0 0.18 0 1 1 1 0 0 x x = 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 14

  15.  A = U Σ V T - example: movie-to-concept Casablanca similarity matrix Serenity Amelie Matrix Alien 0.18 0 SciFi-concept 1 1 1 0 0 0.36 0 2 2 2 0 0 SciFi 9.64 0 0.18 0 1 1 1 0 0 x x = 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 Romnce 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 15

  16. ‘movies’, ‘users’ and ‘concepts’:  U : user-to-concept similarity matrix  V : movie-to-concept sim. matrix  Σ : its diagonal elements: ‘strength’ of each concept 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 16

  17. Movie 2 rating SVD gives best axis to project on:  ‘best’ = min sum first singular of squares of vector projection errors  minimum reconstruction v 1 error Movie 1 rating 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 17

  18.  A = U Σ V T - example: 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 = x x 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 v 1 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 18

  19.  A = U Σ V T - example: variance (‘spread’) on the v 1 axis 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 = x x 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 19

  20.  A = U Σ V T - example:  U Σ: gives the coordinates of the points in the projection axis 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 = x x 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 20

  21. More details  Q: How exactly is dim. reduction done? 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 = x x 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 21

  22. More details  Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 = x x 0 5.29 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 A= 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 22

  23. More details  Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 x x ~ 0 0 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 A= 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 23

  24. More details  Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero: 0.18 0 1 1 1 0 0 0.36 0 2 2 2 0 0 9.64 0 0.18 0 1 1 1 0 0 x x ~ 0 0 5 5 5 0 0 0.90 0 0 0 0 2 2 0 0.53 A= 0 0 0 3 3 0 0.80 0.58 0.58 0.58 0 0 0 0 0 1 1 0 0.27 0 0 0 0.71 0.71 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 24

  25. More details  Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero: 0.18 1 1 1 0 0 0.36 2 2 2 0 0 9.64 0.18 1 1 1 0 0 x x ~ 5 5 5 0 0 0.90 0 0 0 2 2 0 A= 0 0 0 3 3 0 0.58 0.58 0.58 0 0 0 0 0 1 1 0 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 25

  26. More details  Q: How exactly is dim. reduction done?  A: Set the smallest singular values to zero B= 1 1 1 0 0 1 1 1 0 0 2 2 2 0 0 2 2 2 0 0 1 1 1 0 0 1 1 1 0 0 ~ A= 5 5 5 0 0 5 5 5 0 0 0 0 0 0 0 0 0 0 2 2 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 1 1 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 26

  27.  Theorem: Let A = U Σ V T ( σ 1 ≥ σ 2 ≥ …, rank(A)=n) then B = U S V T  S = diagonal n x n matrix where s i = σ i (i=1…k) else s i =0 is a best rank-k approximation to A:  B is solution to min B ǁA -B ǁ F where rank(B)=k  Why? n ∑ − = Σ − = σ − 2 min A B min S min ( s ) s i i = F F i B , rank ( B ) k = i 1 k n n ∑ ∑ ∑ = σ − + σ = σ 2 2 2 min ( s ) s i i i i i = = + = + i 1 i k 1 i k 1 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 27

  28. Equivalent: ‘spectral decomposition’ of the matrix: 1 1 1 0 0 2 2 2 0 0 σ 1 1 1 1 0 0 x x = u 1 u 2 5 5 5 0 0 σ 2 0 0 0 2 2 0 0 0 3 3 v 1 0 0 0 1 1 v 2 1/24/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 28

Recommend

http://cs246.stanford.edu High dimensional == many features Find

http://cs246.stanford.edu High dimensional == many features Find

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu High dimensional == many features Find concepts/topics/genres: Documents: Features: Thousands of words, millions of word pairs

606 views • 57 slides
http://cs246.stanford.edu Input features: N features: X 1 , X 2 , X N A Each X j

http://cs246.stanford.edu Input features: N features: X 1 , X 2 , X N A Each X j

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Input features: N features: X 1 , X 2 , X N A Each X j has domain D j X 1 <v 1 Categorical: C Y= 0.42 D j = {red, blue} X 2

508 views • 40 slides
http://cs246.stanford.edu High dim. High dim. Graph Graph Infinite Infinite Machine Machine Apps

http://cs246.stanford.edu High dim. High dim. Graph Graph Infinite Infinite Machine Machine Apps

3 announcements: Thanks for filling out the HW1 poll HW2 is due today 5pm (scans must be readable) HW3 will be posted today CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu High dim. High dim.

1.07k views • 61 slides
http://cs246.stanford.edu Goal: Given a large number (N in the millions or billions) of text

http://cs246.stanford.edu Goal: Given a large number (N in the millions or billions) of text

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Goal: Given a large number (N in the millions or billions) of text documents, find pairs that are near duplicates Application: Detect

697 views • 68 slides
http://cs246.stanford.edu High dim. Graph Infinite Machine Apps data data data learning

http://cs246.stanford.edu High dim. Graph Infinite Machine Apps data data data learning

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu High dim. Graph Infinite Machine Apps data data data learning Locality Filtering PageRank, Recommen sensitive data SVM SimRank der systems

784 views • 61 slides
http://cs246.stanford.edu Web advertising We discussed how to match

http://cs246.stanford.edu Web advertising We discussed how to match

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Web advertising We discussed how to match advertisers to queries in real-time

686 views • 45 slides
http://cs246.stanford.edu Web pages are not equally important www.joe-schmoe.com vs.

http://cs246.stanford.edu Web pages are not equally important www.joe-schmoe.com vs.

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Web pages are not equally important www.joe-schmoe.com vs. www.stanford.edu We already know: Since there is large diversity in the

668 views • 44 slides
http://cs246.stanford.edu Web advertising We discussed how to match advertisers to

http://cs246.stanford.edu Web advertising We discussed how to match advertisers to

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Web advertising We discussed how to match advertisers to queries in real-time But we did not discuss how to estimate CTR Recommendation

674 views • 45 slides
http://cs246.stanford.edu Classic model of algorithms You get to see the entire input, then

http://cs246.stanford.edu Classic model of algorithms You get to see the entire input, then

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Classic model of algorithms You get to see the entire input, then compute some function of it In this context, offline algorithm

611 views • 39 slides
http://cs246.stanford.edu Web advertising We discussed how to match advertisers to queries

http://cs246.stanford.edu Web advertising We discussed how to match advertisers to queries

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Web advertising We discussed how to match advertisers to queries in real-time But we did not discuss how to estimate the CTR (Click-Through

882 views • 44 slides
http://cs246.stanford.edu Web advertising Weve learned how to match advertisers to

http://cs246.stanford.edu Web advertising Weve learned how to match advertisers to

CS246: Mining Massive Datasets Caroline Lo, Stanford University http://cs246.stanford.edu Web advertising Weve learned how to match advertisers to queries in real-time But how to estimate the CTR (Click-Through Rate)?

815 views • 32 slides
http://cs246.stanford.edu HITS (Hypertext Induced Topic Selection) Is a measure of

http://cs246.stanford.edu HITS (Hypertext Induced Topic Selection) Is a measure of

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu HITS (Hypertext Induced Topic Selection) Is a measure of importance of pages or documents, similar to PageRank Proposed at around same time

855 views • 52 slides
http://cs246.stanford.edu Overlaps with machine learning, statistics, artificial intelligence,

http://cs246.stanford.edu Overlaps with machine learning, statistics, artificial intelligence,

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Overlaps with machine learning, statistics, artificial intelligence, databases, visualization but more stress on scalability of number

613 views • 46 slides
http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom

http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu More algorithms for streams: (1) Filtering a data stream: Bloom filters Select elements with property x from stream (2) Counting distinct

842 views • 40 slides
http://cs246.stanford.edu TAs : Bahman Bahmani Juthika Dabholkar Pierre Kreitmann

http://cs246.stanford.edu TAs : Bahman Bahmani Juthika Dabholkar Pierre Kreitmann

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu TAs : Bahman Bahmani Juthika Dabholkar Pierre Kreitmann Lu Li Aditya Ramesh Office hours: Jure: Tuesdays 9-10am, Gates 418

890 views • 66 slides
http://cs246.stanford.edu Instructor: Jure Leskovec TAs: Aditya Parameswaran

http://cs246.stanford.edu Instructor: Jure Leskovec TAs: Aditya Parameswaran

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Instructor: Jure Leskovec TAs: Aditya Parameswaran Bahman Bahmani Peyman Kazemian 1/3/2011 Jure Leskovec, Stanford C246: Mining

522 views • 35 slides
http://cs246.stanford.edu Rank nodes using link structure PageRank: Link voting: P

http://cs246.stanford.edu Rank nodes using link structure PageRank: Link voting: P

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Rank nodes using link structure PageRank: Link voting: P with importance x has n out links, each link gets x/n votes Page Rs

830 views • 55 slides
http://cs246.stanford.edu Training data 100 million ratings, 480,000 users, 17,770 movies

http://cs246.stanford.edu Training data 100 million ratings, 480,000 users, 17,770 movies

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Training data 100 million ratings, 480,000 users, 17,770 movies 6 years of data: 2000-2005 Test data Last few ratings of each user (2.8

923 views • 53 slides
http://cs246.stanford.edu Supermarket shelf management Market-basket model: Goal: Identify

http://cs246.stanford.edu Supermarket shelf management Market-basket model: Goal: Identify

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Supermarket shelf management Market-basket model: Goal: Identify items that are bought together by sufficiently many customers Approach:

1.24k views • 59 slides
http://cs246.stanford.edu Many real-world problems Web Search and Text Mining Billions

http://cs246.stanford.edu Many real-world problems Web Search and Text Mining Billions

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu Many real-world problems Web Search and Text Mining Billions of documents, millions of terms Product Recommendations Millions of

858 views • 54 slides
http://cs246.stanford.edu 3/9/20 Jure Leskovec, Stanford CS246: Mining Massive Datasets,

http://cs246.stanford.edu 3/9/20 Jure Leskovec, Stanford CS246: Mining Massive Datasets,

Note to other teachers and users of these slides: We would be delighted if you found our material useful for giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a

900 views • 51 slides
http://cs246.stanford.edu It is always possible to decompose a real matrix A into A = U V T ,

http://cs246.stanford.edu It is always possible to decompose a real matrix A into A = U V T ,

Announcements: Submit your project group TODAY (Ed Pinned Post) Project Proposal due this Thursday (no late periods) Upload homework on time (23:59pm)! CS246: Mining Massive Datasets Jure Leskovec, Stanford University

645 views • 49 slides
http://cs246.stanford.edu CPU Machine Learning, Statistics Memory Classical Data Mining

http://cs246.stanford.edu CPU Machine Learning, Statistics Memory Classical Data Mining

CS246: Mining Massive Datasets Jure Leskovec, Stanford University http://cs246.stanford.edu CPU Machine Learning, Statistics Memory Classical Data Mining Disk 1/3/2011 Jure Leskovec, Stanford C246: Mining Massive Datasets 2 20+

1.29k views • 53 slides
http://cs246.stanford.edu Data contains value and knowledge 1/7/20 Jure Leskovec, Stanford

http://cs246.stanford.edu Data contains value and knowledge 1/7/20 Jure Leskovec, Stanford

Note to other teachers and users of these slides: We would be delighted if you found our material useful for giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a

1.21k views • 69 slides

More recommend