Lifting the curse of dimensionality in nonlinear system - PowerPoint PPT Presentation

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Kim Batselier, Zhongming Chen, Ching-Yun Ko, Ngai Wong 1 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction System identification Inputs u ( t ) Outputs y ( t ) System S . . . . . . f ( · ) Main Problem Given measured inputs { u ( t ) ∈ R p } N t =1 , outputs { y ( t ) ∈ R l } N t =1 , and a parametric input-output mapping f ( · ) for the system S , estimate the parameters of f ( · ) . 2 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction NFIR black box model y ( t ) = f ( u ( t ) , u ( t − 1) , . . . , u ( t − M + 1)) with nonlinear function f ( · ) 3 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction NFIR black box model y ( t ) = f ( u ( t ) , u ( t − 1) , . . . , u ( t − M + 1)) with nonlinear function f ( · ) Taylor expansion of f ( · ) : NFIR ⇒ Volterra series 3 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction NFIR black box model y ( t ) = f ( u ( t ) , u ( t − 1) , . . . , u ( t − M + 1)) with nonlinear function f ( · ) Taylor expansion of f ( · ) : NFIR ⇒ Volterra series SISO truncated Volterra series: M − 1 d i � � � y ( t ) = h 0 + h i ( k 1 , . . . , k i ) u ( t − k j ) , i =1 k 1 ,...,k i =0 j =1 3 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction NFIR black box model y ( t ) = f ( u ( t ) , u ( t − 1) , . . . , u ( t − M + 1)) with nonlinear function f ( · ) Taylor expansion of f ( · ) : NFIR ⇒ Volterra series SISO truncated Volterra series: M − 1 d i � � � y ( t ) = h 0 + h i ( k 1 , . . . , k i ) u ( t − k j ) , i =1 k 1 ,...,k i =0 j =1 MIMO truncated Volterra series: each of the l outputs in y ( t ) ∈ R l is a multivariate polynomial in u ( t ) , u ( t − 1) , . . . , u ( t − M + 1) . 3 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction Multivariate polynomials n -variate polynomial of total degree d � d + n � � 7+20 � ≈ n d , e.g. Number of coefficients is = 888030 20 n 4 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction Multivariate polynomials n -variate polynomial of total degree d � d + n � � 7+20 � ≈ n d , e.g. Number of coefficients is = 888030 20 n Curse of dimensionality Truncated Volterra series are described by an exponential amount of unknown coefficients ⇒ need to estimate an exponential amount of coefficients. 4 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction Multivariate polynomials n -variate polynomial of total degree d � d + n � � 7+20 � ≈ n d , e.g. Number of coefficients is = 888030 20 n Curse of dimensionality Truncated Volterra series are described by an exponential amount of unknown coefficients ⇒ need to estimate an exponential amount of coefficients. Main message of this talk Lifting the curse of dimensionality in this system identification problem with tensor networks. 4 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network building blocks a a A A 5 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network building blocks a a A A Three tensor network diagram rules 1 Nodes in the network are tensors 2 Each edge of each node is a particular index of the tensor 3 Connected edges refer to summation over that particular index 5 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 1 C ( i, j ) = � k A ( i, k ) B ( k, j ) 6 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 1 C ( i, j ) = � k A ( i, k ) B ( k, j ) is visually represented by j i k A B 6 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 1 C ( i, j ) = � k A ( i, k ) B ( k, j ) is visually represented by j i k A B Tensor network example 2 k a b 6 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 1 C ( i, j ) = � k A ( i, k ) B ( k, j ) is visually represented by j i k A B Tensor network example 2 k a b � k a ( k ) b ( k ) = a T b 6 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 3 A B D C 7 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 3 A B D C Trace( ABCD ) = Trace( BCDA ) = · · · = Trace( DABC ) 7 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 4 Singular value decomposition of a rank- r matrix A ∈ R p × q : A = U S V T with U ∈ R p × r , V ∈ R q × r orthogonal and S ∈ R r × r diagonal. 8 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Tensor network example 4 Singular value decomposition of a rank- r matrix A ∈ R p × q : A = U S V T with U ∈ R p × r , V ∈ R q × r orthogonal and S ∈ R r × r diagonal. r r U S V T p q 8 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Outer product C = a b T = a ◦ b 9 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Outer product C = a b T = a ◦ b C ( i, j ) = a ( i ) b ( j ) 9 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Outer product C = a b T = a ◦ b C ( i, j ) = a ( i ) b ( j ) 1 � C ( i, j ) = a ( i, k ) b ( k, j ) k =1 9 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Outer product C = a b T = a ◦ b C ( i, j ) = a ( i ) b ( j ) 1 � C ( i, j ) = a ( i, k ) b ( k, j ) k =1 1 a b 9 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks Outer product C = a b T = a ◦ b C ( i, j ) = a ( i ) b ( j ) 1 � C ( i, j ) = a ( i, k ) b ( k, j ) k =1 1 a b Important take-home message Tensor networks with small interconnection dimensions represent tensors with interrelated entries. 9 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network MIMO Volterra model Each of the l outputs in y ( t ) ∈ R l is a multivariate polynomial in u ( t ) , u ( t − 1) , . . . , u ( t − M + 1) . 10 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network MIMO Volterra model Each of the l outputs in y ( t ) ∈ R l is a multivariate polynomial in u ( t ) , u ( t − 1) , . . . , u ( t − M + 1) . Define input and output vectors   y 1 ( t ) .  .   ∈ R l y ( t ) := .  y l ( t ) u ( t − M + 1) T � T ∈ R ( pM +1) � u ( t ) T u t := 1 · · · All input monomials are contained in d � �� � u t ◦ u t ◦ · · · ◦ u t ∈ R ( pM +1) ×···× ( pM +1) 10 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Small example: p = 1 , l = 1 , M = 2 and d = 2   1 u t = u ( t )   u ( t − 1)   1  � � u t ◦ u t = u ( t ) 1 u ( t ) u ( t − 1)  u ( t − 1)   1 u ( t ) u ( t − 1) u ( t ) 2 = u ( t ) u ( t ) u ( t − 1)   u ( t − 1) 2 u ( t − 1) u ( t ) u ( t − 1) 1 u t u t 11 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Symmetric rank-1 input tensor d � �� � u t ◦ u t ◦ · · · ◦ u t ∈ R ( pM +1) ×···× ( pM +1) 1 1 1 u t u t u t pM + 1 pM + 1 pM + 1 12 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Small example: p = 1 , l = 1 , M = 2 and d = 2 y ( t ) = f ( u ( t ) , u ( t − 1)) = v T vec( u t ◦ u t )   1 u ( t )     u ( t − 1)   � �   = v 1 v 2 v 3 v 4 · · · v 9 u ( t )     .   . .   u ( t − 1) 2 13 / 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network MIMO Volterra system y ( t ) = V vec ( u t ◦ u t ◦ · · · ◦ u t ) V ∈ R l × ( pM +1) d contains coefficients of l polynomials 14 / 30