Deterministic Independent Component Analysis (ICA) Ruitong Huang - PowerPoint PPT Presentation

Deterministic Independent Component Analysis (ICA) Ruitong Huang András György Csaba Szepesvári University of Alberta; Imperial College London December 10, 2015 December 10, 2015 1 / 11

Outline 1 Introduction What is ICA, really? 2 Deterministic ICA 3 Conclusions December 10, 2015 2 / 11

What is Independent Component Analysis (ICA)? Introduction What is ICA, really? December 10, 2015 3 / 11

5 BLIND SOURCE SEPARATION 6 4 2 0 − 2 − 4 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 4 2 0 − 2 − 4 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 4 2 0 − 2 − 4 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 Fig. 1.2 The observed signals that are assumed to be mixtures of some underlying source signals. Introduction What is ICA, really? December 10, 2015 4 / 11

3 2 1 0 − 1 − 2 − 3 0 50 100 150 200 250 300 350 400 450 500 2 1.5 1 0.5 0 − 0.5 − 1 − 1.5 − 2 0 50 100 150 200 250 300 350 400 450 500 4 2 0 − 2 − 4 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 Fig. 1.3 The estimates of the original source signals, estimated using only the observed mixture signals in Fig. 1.2. The original signals were found very accurately. Introduction What is ICA, really? December 10, 2015 5 / 11

5 BLIND SOURCE SEPARATION 6 3 4 2 2 0 1 − 2 0 − 4 − 1 − 6 − 2 − 8 0 50 100 150 200 250 300 350 400 450 500 4 − 3 0 50 100 150 200 250 300 350 400 450 500 2 2 1.5 0 1 − 2 0.5 − 4 0 − 0.5 − 6 − 1 − 8 0 50 100 150 200 250 300 350 400 450 500 − 1.5 4 − 2 0 50 100 150 200 250 300 350 400 450 500 2 4 0 2 − 2 0 − 4 − 2 − 6 − 4 − 8 0 50 100 150 200 250 300 350 400 450 500 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 Fig. 1.2 The observed signals that are assumed to be mixtures of some underlying source Fig. 1.3 The estimates of the original source signals, estimated using only the observed signals. mixture signals in Fig. 1.2. The original signals were found very accurately. We were able to estimate the original source signals, using an algorithm that used the information on the independence only. [ . . . ] This leads us to the following definition of ICA [ . . . ] Given a set of observations of random variables ( x 1 ( t ) , x 2 ( t ) , . . . , x d ( t )) , where t is the time or sample index, assume that they are generated as a linear mixture of independent components:     x 1 ( t ) s 1 ( t ) x 2 ( t ) s 2 ( t )      = A  .   .   , . .     . .   x d ( t ) s d ( t ) where A is some unknown matrix. Introduction What is ICA, really? December 10, 2015 6 / 11

Good? Introduction What is ICA, really? December 10, 2015 7 / 11

Independence? 3 2 1 0 − 1 − 2 − 3 0 50 100 150 200 250 300 350 400 450 500 2 1.5 1 0.5 0 − 0.5 − 1 − 1.5 − 2 0 50 100 150 200 250 300 350 400 450 500 4 2 0 − 2 − 4 − 6 − 8 0 50 100 150 200 250 300 350 400 450 500 Fig. 1.3 The estimates of the original source signals, estimated using only the observed mixture signals in Fig. 1.2. The original signals were found very accurately. Is s 1 ( t ) independent of s 2 ( t ) ? Sure! Any two numbers are independent of each other! All deterministic signal sources are fine then? What if s 2 ( t ) = 2 s 1 ( t ) ? Should we be worried about temporal dependencies? No? What if s 1 ( t ) = s 1 ( t + 1 ) = . . . ? Can we redefine ICA in a more meaningful way? Let’s go beyond statistics! Introduction What is ICA, really? December 10, 2015 8 / 11

How to go beyond statistical analysis? 1 Perform a deterministic analysis of the algorithm, reducing the problem to perturbation analysis 2 Perform statistical analysis on the size of perturbations when necessary or desired Let [ T ] = { 1 , . . . , T } . Sources: s : [ T ] → [ − C , C ] d . Let ν ( s ) be the empirical distribution induced by s ; for B ⊂ [ − C , C ] d , ν ( s ) ( B ) = 1 T |{ t ∈ [ T ] : s ( τ ) ∈ B }| . Measure of independence: D 4 ( ν ( s ) , µ ) , D ( d , d ) ( ν ( As ,ǫ ) ) ; 4 Measure of Gaussianness: κ ( ν ( ǫ ) ) ; Measure of Zero-Mean: N ( ν ( ǫ ) ) , N ( ν ( s ) ) Deterministic ICA December 10, 2015 9 / 11

Result There exists a randomized algorithm such that for any A ∈ R d × d , and x , s , ǫ : [ T ] → R d satisfying x ( t ) = As ( t ) + ǫ ( t ) , the algorithm returns ˆ A such that: The computational complexity is O ( d 3 T ) ; With high probability, � D 4 ( ν ( s ) , µ ) + κ ( ν ( ǫ ) ) + D ( d , d ) d ( ˆ ( ν ( As ,ǫ ) ) A , A ) ≤ inf µ ∈ Π 0 C ( µ ) min 4 � + N ( ν ( ǫ ) ) + N ( ν ( s ) ) , Θ( µ ) , Here, C ( µ ) and Θ( µ ) are problem dependent, polynomial in the parameters. Deterministic ICA December 10, 2015 10 / 11

Conclusions ♦ Independent Component Analysis without probabilities! ♦ Deterministic analysis: Cleaner, more general, should do it more often! Limits? ♦ New method: DICA. Universal, strong guarantees. Conclusions December 10, 2015 11 / 11