Characterization of Conditional Independence and Weak Realizations - PowerPoint PPT Presentation

Characterization of Conditional Independence and Weak Realizations of Multivariate Gaussian Random Variables: Applications to Networks Charalambos D. Charalambous and Jan H. van Schuppen 2020 IEEE International Symposium on Information Theory 21-26 June 2020 Los Angeles, California, USA

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Problems of Multi-User Communication Problem. We consider the Gray and Wyner 1 lossy network coding for a tuple of multivariate jointly Gaussian random variables (RVs): ( X N 2 ) = {( X 1 , i , X 2 , i ) ∶ i = 1 , 2 ,..., N } , 1 , X N (1) X 1 , i ∶ Ω → R p 1 = X 1 , X 2 , i ∶ Ω → R p 2 = X 2 , ∀ i , (2) P X 1 , i X 2 , i = P X 1 , X 2 jointly Gaussian and ( X 1 , i , X 2 , i ) indep. of ( X 1 , j , X 2 , j ) , ∀ i ≠ j (3) with average square-error distortions at the two decoders i )} ≤ ∆ i , ∆ i ∈ [ 0 , ∞ ] , i = 1 , 2 , E { D X i ( X N i , ˆ X N (4) D X i ( x N i ) ∣∣ x i , j − ˆ x i , j ∣∣ 2 ∑ N = 1 △ R pi , i = 1 , 2 , x N i , ˆ (5) N j = 1 1 R. M. Gray and A.D. Wyner, “Source coding for a simple network”, Bell Systems Technical Journal, 1974

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Problems of Multi-User Communication (cont) Main Results of ISIT Paper: Calculation of the lossy common information, i.e, minimum common message rate on the Gray and Wyner rate region, when the sum rate is equal to the joint rate distortion function (RDF) Parametrization of Gray and Wyner rate region and rates that lie on the Pangloss plane Weak stochastic realization of RVs that achieve the rates, and induce optimal test channels of conditional, marginal, and joint RDFs

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Problems of Multi-User Communication Technical Tools-Three Concepts: Hotelling’s geometric approach to Gaussian RVs van Putten’s and van Schuppen’s parametrization of the family of all Gaussian distributions P X 1 , X 2 , W by an auxiliary Gaussian RV W ∶ Ω → R k = W that makes X 1 and X 2 conditional independent Weak stochastic realization of ( X 1 , X 2 , W ) Additional Applications not Discussed: Capacity of the multiple access channel Secrecy capacity, etc

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Notation An R n -valued Gaussian RV, X ∶ Ω → R n is denoted by X ∈ G ( m X , Q X ) , with as parameters the mean value m X ∈ R n and the variance Q X ∈ R n × n , Q X = Q T X ⪰ 0. The effective dimension of the RV X ∈ G ( 0 , Q X ) is denoted by dim ( X ) = rank ( Q X ) . An n × n identity matrix is denoted by I n . A tuple of Gaussian RVs ( X 1 , X 2 ) ∈ G ( 0 , Q ( X 1 , X 2 ) ) will be denoted this way to save space, rather than by ( X 1 X 2 ) , Q ( X 1 , X 2 ) = ( Q X 1 ) ∈ R ( p 1 + p 2 )×( p 1 + p 2 ) . Q X 1 , X 2 Q T Q X 2 X 1 , X 2

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks The Gray and Wyner Lossy Rate Region R GW ( ∆ 1 , ∆ 2 ) denotes the Gray and Wyner operational rate region , by a coding scheme that uses the auxiliary RV W ∶ Ω → W . Define the family of probability distributions P ≜{ P X 1 , X 2 , W , x 1 ∈ X 1 , x 2 ∈ X 2 , w ∈ W ∣ } P X 1 , X 2 , W ( x 1 , x 2 , ∞) = P X 1 , X 2 Theorem 8 in Gray and Wyner 1974 (Under conditions). For each P X 1 , X 2 , W ∈ P and ∆ 1 ≥ 0 , ∆ 2 ≥ 0, define R ( ∆ 1 , ∆ 2 ) = {( R 0 , R 1 , R 2 ) ∶ R 0 ≥ I ( X 1 , X 2 ; W ) , P X 1 , X 2 , W GW R 1 ≥ R X 1 ∣ W ( ∆ 1 ) , R 2 ≥ R X 2 ∣ W ( ∆ 2 )} (6) where R X i ∣ W ( ∆ i ) is the conditional RDF of X i , conditioned on W .

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks The Gray and Wyner Lossy Rate Region Let R ∗ GW ( ∆ 1 , ∆ 2 ) = ( R ( ∆ 1 , ∆ 2 )) ⋃ c P X 1 , X 2 , W △ (7) GW P X 1 , X 2 , W ∈P The achievable rate region is R GW ( ∆ 1 , ∆ 2 ) = R ∗ GW ( ∆ 1 , ∆ 2 ) . (8) Theorem 6 in Gray and Wyner 1974. if ( R 0 , R 1 , R 2 ) ∈ R GW ( ∆ 1 , ∆ 2 ) , then R 0 + R 1 + R 2 ≥ R X 1 , X 2 ( ∆ 1 , ∆ 2 ) , called pangloss bound (9) R 0 + R 1 ≥ R X 1 ( ∆ 1 ) , R 0 + R 2 ≥ R X 2 ( ∆ 2 ) (10) Pangloss Plane: the set of triples ( R 0 , R 1 , R 2 ) ∈ R GW ( ∆ 1 , ∆ 2 ) that satisfy R 0 + R 1 + R 2 = R X 1 , X 2 ( ∆ 1 , ∆ 2 )

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Wyner’s Common Information Wyner’s characterization of common information 2 (for finite-valued RVs): C W ( X 1 , X 2 ) P X 1 , X 2 , W ∶ P X 1 , X 2 ∣ W = P X 1 ∣ W P X 2 ∣ W I ( X 1 , X 2 ; W ) △ = inf (11) C W ( X 1 , X 2 ) does not have an operational meaning for continuous-valued RVs, such as, Gaussian C W ( X 1 , X 2 ) is a single point on the Gray and Wyner rate regions (for finite-valued RVs) 2 A.D. Wyner, “The common information of two dependent random variables”, IEEEIT, 1975

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Wyner’s Lossy Common Information Wyner’s characterization of lossy common information 3 : Let C GW ( X 1 , X 2 ;∆ 1 , ∆ 2 ) denote the minimum common message rate R 0 on R GW ( ∆ 1 , ∆ 2 ) , with sum rate not exceeding the joint RDF R X 1 , X 2 ( ∆ 1 , ∆ 2 ) . Then C GW ( X 1 , X 2 ;∆ 1 , ∆ 2 ) = inf I ( X 1 , X 2 ; W ) , such that (12) R X 1 ∣ W ( ∆ 1 ) + R X 2 ∣ W ( ∆ 2 ) + I ( X 1 , X 2 ; W ) = R X 1 , X 2 ( ∆ 1 , ∆ 2 ) (13) The infimum is over all RVs W in W , which parametrize the source distribution via P X 1 , X 2 , W , having a X 1 × X 2 − marginal source distribution P X 1 , X 2 , and induce joint distributions P W , X 1 , X 2 , ˆ X 2 . X 1 , ˆ 3 K. B. Viswanatha, E. Akyol, K. Rose, “The lossy common information of correlated sources”, IEEEIT, 2014, G. Xu, W. Liu, B. Chen, “A lossy source coding interpretation of Wyner’s common information”, IEEEIT, 2016

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks Common, Correlated, Private Parts of ( X 1 , X 2 ) Hotelling’s 1936 geometric approach to Gaussian RVs: The geometric object of a Gaussian RV Y ∶ Ω → R p is the σ − algebra F Y generated by Y . For ( X 1 , X 2 ) ∈ G ( 0 , Q ( X 1 , X 2 ) ) , a basis transformation consists of a non-singular matrix, S , = Block-diag ( S 1 , S 2 ) , X c = S 1 X , X c = S 2 X 2 , △ △ △ (14) S 1 2 F X 1 = F S 1 X 1 , F X 2 = F S 2 X 2 . (15) S maps ( X 1 , X 2 ) into the “canonical form” of the tuple of RVs Full specification 4 , as interpreted in the table below. 4 C. D. Charalambous and J. H. van Schuppen, “A New Approach to Lossy Network Compression of a Tuple of Correlated Multivariate Gaussian RVs”, 29 May 2019, https://arxiv.org/abs/1905.12695

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks 21 − a . s . 11 = X c X c identical information of X c 1 and X c 2 X c correlated information of X c 1 w.r.t X c 12 2 X c private information of X c 1 w.r.t X c 11 − a . s . 13 2 21 = X c X c identical information of X c 1 and X c 2 X c correlated information of X c 2 w.r.t X c 22 1 X c private information of X c 2 w.r.t X c 23 1 ij ∶ Ω → R p ij , i = 1 , 2 , j = 1 , 2 , 3 , p 11 = p 21 , p 12 = p 22 = n , (16) X c p 1 = p 11 + p 12 + p 13 , p 2 = p 21 + p 22 + p 23 , (17) S 1 X 1 = ( X c 13 ) , S 2 X 2 = ( X c 23 ) , 11 , X c 12 , X c 21 , X c 22 , X c (18) 21 − a . s ., X c 21 ∈ G ( 0 , I p 11 ) , 11 = X c X c 11 , X c (19) 13 ∈ G ( 0 , I p 13 ) and X c 23 ∈ G ( 0 , I p 23 ) are independent X c (20) 12 ∈ G ( 0 , I p 12 ) and X c 22 ∈ G ( 0 , I p 22 ) are correlated , X c (21) E [ X c 12 ( X c 22 ) T ] = D = Diag ( d 1 ,..., d p 12 ) , d i ∈ ( 0 , 1 ) ∀ i . (22) Entries of D are called the canonical correlation coefficients (23)

Motivation Notation The Gray and Wyner Rate Region Main Concepts Our Results Concluding Remarks van Putten’s and van Schuppen’s parametrization of conditional independence van Putten’s and van Schuppen’s parametrization 5 : Define the family of all jointly Gaussian probability distributions P X 1 , X 2 , W by a Gaussian RV W ∶ Ω → R k = W that makes X 1 and X 2 conditional independent, P CIG △ = { P X 1 , X 2 , W ∣ P X 1 , X 2 ∣ W = P X 1 ∣ W P X 2 ∣ W , the X 1 × X 2 − marginal dist. of P X 1 , X 2 , W is the fixed dist. P X 1 , X 2 , and ( X 1 , X 2 , W ) is jointly Gaussian } P CIG min ⊆ P CIG additionally the dimension of RV W is minimal 5 C. van Putten and J.H. van Schuppen, “The weak and strong Gaussian probabilistic realization problem”, J. Multivariate Anal., 1985