tensor factorization via matrix factorization
play

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov - PowerPoint PPT Presentation

Jun 12, 2023 •303 likes •1.1k views

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy Liang Stanford University May 11, 2015 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 1 / 28 Introduction:

  1. Tensor Factorization via Matrix Factorization Volodymyr Kuleshov ú Arun Tejasvi Chaganty ú Percy Liang Stanford University May 11, 2015 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 1 / 28

  2. Introduction: tensor factorization An application: community detection a b c d Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 2 / 28

  3. Introduction: tensor factorization An application: community detection a b c d Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 2 / 28

  4. Introduction: tensor factorization An application: community detection a b c d Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 2 / 28

  5. Introduction: tensor factorization An application: community detection ? ? a b ? ? c d Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 2 / 28

  6. Introduction: tensor factorization An application: community detection Anandkumar, Ge, Hsu, and S. Kakade 2013 a b c d = + + · · · + Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 2 / 28

  7. Introduction: tensor factorization Applications of tensor factorization I Community detection I Anandkumar, Ge, Hsu, and S. Kakade 2013 I Parsing I Cohen, Satta, and Collins 2013 I Knowledge base completion I Chang et al. 2014 I Singh, Rockt¨ aschel, and Riedel 2015 I Topic modelling I Anandkumar, Foster, et al. 2012 I Crowdsourcing I Zhang et al. 2014 I Mixture models I Anandkumar, Ge, Hsu, S. M. Kakade, et al. 2013 I Bottlenecked models I Chaganty and Liang 2014 I . . . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 3 / 28

  8. Introduction: tensor factorization What is tensor (CP) factorization? I Tensor analogue of matrix eigen-decomposition. k ÿ M = fi i u i ¢ u i . i =1 = + + · · · + k Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 4 / 28

  9. Introduction: tensor factorization What is tensor (CP) factorization? I Tensor analogue of matrix eigen-decomposition. k ÿ T = fi i u i ¢ u i ¢ u i . i =1 = + + · · · + k Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 4 / 28

  10. Introduction: tensor factorization What is tensor (CP) factorization? I Tensor analogue of matrix eigen-decomposition. k ÿ ‚ T = fi i u i ¢ u i ¢ u i + ‘ R . i =1 I Goal: Given T with noise, ‘ R , recover factors u i . = + + · · · + + k Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 4 / 28

  11. Introduction: tensor factorization What is tensor (CP) factorization? I Tensor analogue of matrix eigen-decomposition. k ÿ ‚ T = fi i u i ¢ u i ¢ u i + ‘ R . i =1 I Goal: Given T with noise, ‘ R , recover factors u i . l a = n + + · · · + + o g o h t r O l a n o = + + · · · + + g o h t r o - n o N Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 4 / 28

  12. Introduction: tensor factorization Existing tensor factorization algorithms I Tensor power method (Anandkumar, Ge, Hsu, S. M. Kakade, et al. 2013) I Analog of matrix power method. I Sensitive to noise. I Restricted to orthogonal tensors. Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 5 / 28

  13. Introduction: tensor factorization Existing tensor factorization algorithms I Tensor power method (Anandkumar, Ge, Hsu, S. M. Kakade, et al. 2013) I Analog of matrix power method. I Sensitive to noise. I Restricted to orthogonal tensors. I Alternating least squares (Comon, Luciani, and Almeida 2009; Anandkumar, Ge, and Janzamin 2014) I Sensitive to initialization. Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 5 / 28

  14. Introduction: tensor factorization Existing tensor factorization algorithms I Tensor power method (Anandkumar, Ge, Hsu, S. M. Kakade, et al. 2013) I Analog of matrix power method. I Sensitive to noise. I Restricted to orthogonal tensors. I Alternating least squares (Comon, Luciani, and Almeida 2009; Anandkumar, Ge, and Janzamin 2014) I Sensitive to initialization. Our approach: reduce to existing fast and robust matrix algorithms . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 5 / 28

  15. Orthogonal Tensor factorization Outline Introduction: tensor factorization Orthogonal Tensor factorization Projections Non-orthogonal tensor factorization Related work Empirical results Conclusions Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 6 / 28

  16. Orthogonal Tensor factorization Tensor factorization via single matrix factorization fi 1 u ¢ 3 fi 2 u ¢ 3 fi 3 u ¢ 3 T = + + + ‘ R 1 2 3 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 7 / 28

  17. Orthogonal Tensor factorization Tensor factorization via single matrix factorization u ¢ 3 u ¢ 3 u ¢ 3 T = + + 1 1 1 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 8 / 28

  18. Orthogonal Tensor factorization Tensor factorization via single matrix factorization u ¢ 3 u ¢ 3 u ¢ 3 T = + + 1 1 1 ¿ ( w € u 1 ) u ¢ 2 ( w € u 2 ) u ¢ 2 ( w € u 3 ) u ¢ 2 T ( I , I , w ) = + + 1 2 3 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 8 / 28

  19. Orthogonal Tensor factorization Tensor factorization via single matrix factorization u ¢ 3 u ¢ 3 u ¢ 3 T = + + 1 1 1 ¿ ( w € u 1 ) u ¢ 2 ( w € u 2 ) u ¢ 2 ( w € u 3 ) u ¢ 2 T ( I , I , w ) = + + 1 2 3 ¸ ˚˙ ˝ ¸ ˚˙ ˝ ¸ ˚˙ ˝ ⁄ 1 ⁄ 2 ⁄ 3 I Proposal: Eigen-decomposition on the projected matrix. I Return: recovered eigenvectors, u i . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 8 / 28

  20. Orthogonal Tensor factorization Sensitivity of single matrix projection I Problem : Eigendecomposition is very sensitive to the eigengap . 1 error in factors à min(di ff erence in eigenvalues) . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 9 / 28

  21. Orthogonal Tensor factorization Sensitivity of single matrix projection I Problem : Eigendecomposition is very sensitive to the eigengap . 1 error in factors à min(di ff erence in eigenvalues) . I Intuition: If two eigenvalues are equal, corresponding eigenvectors are arbitrary. = + + Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 9 / 28

  22. Orthogonal Tensor factorization Sensitivity of single matrix projection I Problem : Eigendecomposition is very sensitive to the eigengap . 1 error in factors à min(di ff erence in eigenvalues) . I Intuition: If two eigenvalues are equal, corresponding eigenvectors are arbitrary. = + + Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 9 / 28

  23. Orthogonal Tensor factorization Sensitivity analysis (contd.) Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  24. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  25. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  26. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  27. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  28. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  29. Orthogonal Tensor factorization Sensitivity analysis (contd.) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  30. Orthogonal Tensor factorization Sensitivity analysis (contd.) (Cardoso 1994) I Single matrix factorization: error in factors à 1 min di ff . in eigenvalues . I Simultaneous matrix factorization: error in factors à 1 min avg. di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

  31. Orthogonal Tensor factorization Sensitivity analysis (contd.) (Cardoso 1994) I Single matrix factorization: Every coordinate pair needs one good projection (with a error in factors à large eigengap). 1 min di ff . in eigenvalues . I Simultaneous matrix factorization: error in factors à 1 min avg. di ff . in eigenvalues . Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 10 / 28

Recommend

L101: Matrix Factorization In a nutshell Matrix factorization/completion you know? In NLP?

L101: Matrix Factorization In a nutshell Matrix factorization/completion you know? In NLP?

L101: Matrix Factorization In a nutshell Matrix factorization/completion you know? In NLP? Word embeddings Topic models Information extraction FastText Why complete the matrix? Label Features Label Label Label Label

420 views • 20 slides
Online-Updating Regularized Kernel Matrix Factorization Models for Large-Scale Recommender

Online-Updating Regularized Kernel Matrix Factorization Models for Large-Scale Recommender

Introduction Matrix factorization Learning Matrix Factorization Models Online Updates Evaluation Online-Updating Regularized Kernel Matrix Factorization Models for Large-Scale Recommender Systems. Steffen Rendle, Lars Schmidt-Thieme

903 views • 21 slides
Matrix Factorization and Factorization Machines for Recommender Systems Chih-Jen Lin Department

Matrix Factorization and Factorization Machines for Recommender Systems Chih-Jen Lin Department

Matrix Factorization and Factorization Machines for Recommender Systems Chih-Jen Lin Department of Computer Science National Taiwan University Talk at SDM workshop on Machine Learning Methods on Recommender Systems, May 2, 2015 Chih-Jen Lin

1.14k views • 54 slides
Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms,

844 views • 80 slides
TIME-DEPENDENT PARAMETRIC AND HARMONIC TEMPLATES IN NON-NEGATIVE MATRIX FACTORIZATION 13 th

TIME-DEPENDENT PARAMETRIC AND HARMONIC TEMPLATES IN NON-NEGATIVE MATRIX FACTORIZATION 13 th

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion TIME-DEPENDENT PARAMETRIC AND HARMONIC TEMPLATES IN NON-NEGATIVE MATRIX FACTORIZATION 13 th International Conference on Digital

750 views • 26 slides
Fast Coordinate Descent methods for Non-Negative Matrix Factorization Inderjit S. Dhillon

Fast Coordinate Descent methods for Non-Negative Matrix Factorization Inderjit S. Dhillon

Non-negative Matrix Factorization Fast Coordinate Descent methods for Non-Negative Matrix Factorization Inderjit S. Dhillon University of Texas at Austin SIAM Conference on Applied Linear Algebra Valencia, Spain June 19, 2012 Joint work with

682 views • 25 slides
Matrix Factorization with Binary Components Uniqueness in a randomized model Felix Krahmer,

Matrix Factorization with Binary Components Uniqueness in a randomized model Felix Krahmer,

Matrix Factorization with Binary Components Uniqueness in a randomized model Felix Krahmer, TU M unchen Joint work with: Matthias Hein, Saarland University, David James , University of G ottingen Matrix Factorization given data

1.49k views • 44 slides
Matrix Factorization For Topic Models Dr. Derek Greene Insight Latent Space Workshop

Matrix Factorization For Topic Models Dr. Derek Greene Insight Latent Space Workshop

Matrix Factorization For Topic Models Dr. Derek Greene Insight Latent Space Workshop Non-negative Matrix Factorization NMF : an unsupervised family of algorithms that simultaneously perform dimension reduction and clustering. Also

407 views • 16 slides
Factorization on GPUs Wei Tan IBM T. J. Watson Research Center wtan@us.ibm.com

Factorization on GPUs Wei Tan IBM T. J. Watson Research Center wtan@us.ibm.com

cuMF_sgd: Fast and Scalable Matrix Factorization on GPUs Wei Tan IBM T. J. Watson Research Center wtan@us.ibm.com http://github.com/cumf http://researcher.ibm.com/person/us-wtan GTC 2017 Agenda Why accelerate matrix factorization?

571 views • 21 slides
A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Medium-Grained Sparse Tensor Factorization 1 / 24

424 views • 25 slides
CuMF: Large-Scale Matrix Factorization on Just One Machine with GPUs Wei Tan, IBM T. J. Watson

CuMF: Large-Scale Matrix Factorization on Just One Machine with GPUs Wei Tan, IBM T. J. Watson

CuMF: Large-Scale Matrix Factorization on Just One Machine with GPUs Wei Tan, IBM T. J. Watson Research Center wtan@us.ibm.com Agenda Why accelerate recommendation (matrix factorization) using GPUs? What are the challenges? How cuMF

403 views • 22 slides
A Model For Mixed Linear-Tropical Matrix Factorization James Hook, Sanjar Karaev, Pauli Miettinen

A Model For Mixed Linear-Tropical Matrix Factorization James Hook, Sanjar Karaev, Pauli Miettinen

A Model For Mixed Linear-Tropical Matrix Factorization James Hook, Sanjar Karaev, Pauli Miettinen University of Birmingham: 18th June 2018 Low-Rank Approximate Factorization Given a matrix A R n m , an approximate factorization of rank k

207 views • 17 slides
Greedy Orthogonal Pivoting for Non-negative Matrix Factorization Kai Zhang, Jun Liu, Jie Zhang,

Greedy Orthogonal Pivoting for Non-negative Matrix Factorization Kai Zhang, Jun Liu, Jie Zhang,

Greedy Orthogonal Pivoting for Non-negative Matrix Factorization Kai Zhang, Jun Liu, Jie Zhang, Jun Wang Infinia ML Inc., Fudan University, East China Normal University Non-negative Matrix Factorization Represent data with non-negative basis

330 views • 7 slides
Matrix Factorization and Lifting Palle Jorgensen and Myung-Sin Song The University of Iowa,

Matrix Factorization and Lifting Palle Jorgensen and Myung-Sin Song The University of Iowa,

Matrix Factorization and Lifting Palle Jorgensen and Myung-Sin Song The University of Iowa, Southern Illinois University Edwardsville January 7, 2011 Introduction Develop and refine a procedure which uses factorization of families of

544 views • 35 slides
Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown University Instructor: Ellie

Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown University Instructor: Ellie

Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown University Instructor: Ellie Pavlick HTAs: Josh Levin, Diane Mutako, Sol Zitter 1 Announcements 2 Today Matrix Factorization with SVD Applications to: Topic

1.33k views • 117 slides
Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals

Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals

Infinite Positive Semidefinite Tensor Factorization by K. Yoshii, R. Tomioka, D. Mochihashi, and M. Goto Infinite Positive Semidefinite Tensor Factorization for Source Separation of Mixture Signals Kazuyoshi Yoshii Ryota Tomioka Daichi

461 views • 8 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla December 17, 2011 Peter J. Haas Yannis Sismanis Christina Teflioudi Faraz Makari Outline Matrix Factorization Stochastic Gradient Descent

1.24k views • 75 slides
Chapter IX: Matrix factorizations* 1. The general idea 2. Matrix factorization methods 3. Latent

Chapter IX: Matrix factorizations* 1. The general idea 2. Matrix factorization methods 3. Latent

Chapter IX: Matrix factorizations* 1. The general idea 2. Matrix factorization methods 3. Latent topic models 4. Dimensionality reduction *Zaki & Meira, Ch. 8; Tan, Steinbach & Kumar, App. B; Manning, Raghavan & Schtze, Ch. 18

798 views • 53 slides
Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla

Large-Scale Matrix Factorization with Distributed Stochastic Gradient Descent Rainer Gemulla August 23, 2011 Peter J. Haas Yannis Sismanis Erik Nijkamp Outline Matrix Factorization Stochastic Gradient Descent Distributed SGD with MapReduce

1.05k views • 81 slides
Towards Faster Nonnegative Tensor Factorization: A New Active-Set type Algorithm and Comparisons

Towards Faster Nonnegative Tensor Factorization: A New Active-Set type Algorithm and Comparisons

Towards Faster Nonnegative Tensor Factorization: A New Active-Set type Algorithm and Comparisons Haesun Park hpark@cc.gatech.edu College of Computing Georgia Institute of Technology Atlanta, GA 30332, USA Joint work with Krishnakumar

502 views • 24 slides
Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole Normale Sup erieure March 2011 Joint work with R. Jenatton, J. Mairal, G. Obozinski Structured sparse methods for matrix factorization Outline

1.07k views • 73 slides
Divide-and-Conquer Matrix Factorization Lester Mackey Collaborators: Ameet Talwalkar

Divide-and-Conquer Matrix Factorization Lester Mackey Collaborators: Ameet Talwalkar

Divide-and-Conquer Matrix Factorization Lester Mackey Collaborators: Ameet Talwalkar Michael I. Jordan Stanford University UCLA UC Berkeley December 14, 2015 Mackey (Stanford) Divide-and-Conquer Matrix Factorization

891 views • 42 slides
Multimodal Visualization Based On Non-negative Matrix Factorization Jorge Camargo Juan Caicedo

Multimodal Visualization Based On Non-negative Matrix Factorization Jorge Camargo Juan Caicedo

Multimodal Visualization Based On Non-negative Matrix Factorization Multimodal Visualization Based On Non-negative Matrix Factorization Jorge Camargo Juan Caicedo Fabio Gonzlez BioIngenium Research Group National University of Colombia

406 views • 29 slides
Word Semantic Representations using Bayesian Probabilistic Tensor Factorization Jingwei Zhang,

Word Semantic Representations using Bayesian Probabilistic Tensor Factorization Jingwei Zhang,

Word Semantic Representations using Bayesian Probabilistic Tensor Factorization Jingwei Zhang, Jeremy Salwen, Michael Glass and Alfio Gliozzo Department of Computer Science Columbia University IBM T.J. Watson Research Center Tuesday 21 st

512 views • 33 slides

More recommend