Relation Regularized Matrix Factorization Wu-Jun Li, Dit-Yan Yeung - PowerPoint PPT Presentation

Relation Regularized Matrix Factorization Wu-Jun Li, Dit-Yan Yeung Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong, China IJCAI 2009 Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 1 / 23

Contents 1 Introduction 2 Relation Regularized Matrix Factorization Model Formulation Learning Convergence and Complexity Analysis 3 Experiments 4 Conclusion Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 2 / 23

Introduction Matrix Factorization (MF) To project instances into a lower-dimensional latent space. X : n × m , with each row X i ∗ denoting an instance X ≈ UV T U : n × D V : m × D D < m U i ∗ is the lower-dimensional representation of X i ∗ Objective: To get a U which can remove the noise in X Example: Latent semantic indexing (LSI) for document analysis Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 3 / 23

Introduction Relational Data Contain both content information and relation (link) structure. Examples: Web pages: page content and hyperlinks Research papers: paper content and citations Representation: two matrices Content matrix Link matrix Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 4 / 23

Introduction Semantics of Relations There exist at least two types of links with different semantics: Type I Links: If two instances link to or are linked by one common instance, they will be most likely to belong to the same class. V1 V1 V3 V3 V2 V2 Example: Hyperlinks among web pages Type II Links: Two linked instances are most likely to belong to the same class. V1 V1 V1 V3 V3 V3 V2 V2 V2 Example: Citations among research papers Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 5 / 23

Introduction Existing Work Traditional MF methods: Can only model one matrix Example: LSI Joint Link-Content MF (LCMF): Can model both content and link matrices simultaneously Can only model Type I links Illustration: Link structure Result of LCMF -.8 -.5 .3 -.1 -.0 V3 V5 V7 -.0 .4 .6 -.1 -.4 -.0 .4 .6 -.1 -.4 .3 -.2 .3 -.4 .3 V1 V8 .3 -.2 .3 -.4 .3 -.4 .5 .0 -.2 .6 -.4 .5 .0 -.2 .6 V2 V4 V6 -.1 .1 -.4 -.8 -.4 Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 6 / 23

Introduction Our Contribution Relation regularized matrix factorization (RRMF): To model Type II links Can also model Type I links by preprocessing the link structure Convergent Linear time-complexity Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 7 / 23

Relation Regularized Matrix Factorization Model Formulation Notations Content matrix: X − n × m X i ∗ : content feature vector for instance i Adjacency matrix: A − n × n A ij = 1 if there is a relation between instances i and j , and otherwise A ij = 0; A ii = 0 Note: This specification of A is only suitable for Type II links. We will introduce the strategy to specify A for Type I links later. Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 8 / 23

Relation Regularized Matrix Factorization Model Formulation Objective Function 1 2 � X − UV T � 2 + α 2 ( � U � 2 + � V � 2 ) + β 2 tr ( U T L U ) min U , V where L = D − A and D is a diagonal matrix with D ii = � j A ij . n n 1 � � tr ( U T L U ) A ij � U i ∗ − U j ∗ � 2 = 2 i =1 j =1 The goal of tr ( U T L U ) is to make the latent representations of two instances as close as possible if there exists a relation between them. ⇒ in line with the semantics of Type II links Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 9 / 23

Relation Regularized Matrix Factorization Model Formulation Illustration The original feature representation and link structure: The goal to be achieved by RRMF: Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 10 / 23

Relation Regularized Matrix Factorization Model Formulation Adapting RRMF for Type I Links Basic idea: To transform Type I links to Type II links Strategy: Artificially add a link between two instances if they link to or are linked by a common instance. V1 V1 V3 V3 V2 V2 Type I links Type II links Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 11 / 23

Relation Regularized Matrix Factorization Learning Convexity of the Objective Function 2 � X − UV T � 2 + α 2 ( � U � 2 + � V � 2 ) + β Let f = 1 2 tr ( U T L U ). Theorem f is convex w.r.t. U . Theorem f is convex w.r.t. V . Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 12 / 23

Relation Regularized Matrix Factorization Learning Alternating Projection Method Each time we fix one parameter and then update the other one. Iterate until some termination condition is satisfied: Learn U with V fixed Learn V with U fixed Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 13 / 23

Relation Regularized Matrix Factorization Learning Learning U To optimize one column U ∗ d at a time with the other columns fixed: F ( d ) U ∗ d = e ( d ) E ( d ) + α I + β L F ( d ) = m ∂ 2 g ∂ 2 g ∂ 2 g ∂ 2 g E ( d ) � V 2 = diag( ) , = , , . . . , jd ∂ U 1 d ∂ U 1 d ∂ U 2 d ∂ U 2 d ∂ U nd ∂ U nd ∂ U id ∂ U id j =1 m ( e ( d ) 1 , e ( d ) 2 , . . . , e ( d ) n ) T , e ( d ) e ( d ) � V jd ( X ij − U i ∗ V T = = j ∗ + U id V jd ) i j =1 Steepest descent to iteratively update U ∗ d : e ( d ) − F ( d ) U ∗ d ( t ) r ( t ) = r ( t ) T r ( t ) δ ( t ) = r ( t ) T F ( d ) r ( t ) U ∗ d ( t + 1) = U ∗ d ( t ) + δ ( t ) r ( t ) Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 14 / 23

Relation Regularized Matrix Factorization Learning Learning V The update of the whole matrix V can naturally be decomposed into the update of each row V j ∗ : � n � � K − 1 V j ∗ = X ij U i ∗ i =1 n � U T K = i ∗ U i ∗ + α I i =1 Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 15 / 23

Relation Regularized Matrix Factorization Convergence and Complexity Analysis Convergence and Complexity Theorem The learning algorithm will converge. The time complexity of the learning algorithm is O ( n ). Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 16 / 23

Experiments Data Sets WebKB: Web pages from the CS departments of 4 universities: Cornell, Texas, Washington, Wisconsin 7-class problem: a page belongs to one of the { student, professor, course, project, staff, department or “others” } . Cora: Research papers with their bibliographic citations Each paper is labeled as one of the subfields of data structure (DS), hardware and architecture (HA), machine learning (ML), and programming language (PL). Characteristics of the WebKB data set Characteristics of the Cora data set #classes #instances #terms #classes #instances #terms Cornell 7 827 4,134 DS 9 751 6,234 Texas 7 814 4,029 HA 7 400 3,989 Washington 7 1,166 4,165 ML 7 1,617 8,329 Wisconsin 6 1,210 4,189 PL 9 1,575 7,949 Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 17 / 23

Experiments Convergence Speed 5 x 10 12 0.9 Objective Function Value 10 0.85 8 Accuracy 0.8 6 0.75 4 0.7 2 0.65 0 0 10 20 30 40 50 0 10 20 30 40 50 T T (a) Objective function (b) Accuracy Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 18 / 23

Experiments Performance on Cora 90 85 SVM on content 80 SVM on links Accuracy (in %) SVM on link−content 75 directed graph regularization PLSI+PHITS 70 PCA 65 MMMF link−content MF 60 link−content sup. MF RRMF 55 50 45 DS HA ML PL Data Set Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 19 / 23

Experiments Performance on WebKB 100 95 SVM on content 90 SVM on links SVM on link−content Accuracy (in %) directed graph regularization 85 PLSI+PHITS PCA 80 MMMF link−content MF 75 link−content sup. MF RRMF 70 65 Cornell Texas Washington Wisconsin Data Set Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 20 / 23

Experiments Sensitivity to Parameters 95 90 90 85 Accuracy (in %) Accuracy (in %) 80 85 75 70 80 65 60 10 20 30 40 50 0 50 100 150 200 D beta (a) Effect of β (b) Effect of D Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 21 / 23

Conclusion Main Contributions RRMF seamlessly integrates both relation and content information RRMF achieves state-of-the-art performance. RRMF is scalable to large-scale problems. RRMF is convergent and very stable. Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 22 / 23

Conclusion Code MATLAB code and data can be downloaded from: http://www.cse.ust.hk/ ∼ liwujun Li and Yeung (CSE, HKUST) RRMF IJCAI 2009 23 / 23