Bounding Techniques for the Intrinsic Uncertainty of Channels Or Ordentlich Joint work with Ofer Shayevitz July 4th, 2014 ISIT, Honolulu, HI, USA Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation DMC For DMCs C = max P ( X ) I ( X ; Y ) Calculating I ( X ; Y ) is easy Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation DMC For DMCs C = max P ( X ) I ( X ; Y ) Calculating I ( X ; Y ) is easy Channels with memory Assuming information stability [Dobrushin 1973] 1 C = lim n →∞ max nI ( X ; Y ) P X Calculating I ( X ; Y ) may be difficult Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation DMC For DMCs C = max P ( X ) I ( X ; Y ) Calculating I ( X ; Y ) is easy Channels with memory Assuming information stability [Dobrushin 1973] 1 C = lim n →∞ max nI ( X ; Y ) P X Calculating I ( X ; Y ) may be difficult For many interesting channels P XY has sparse support: -deletion, insertion, trapdoor,... Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation DMC For DMCs C = max P ( X ) I ( X ; Y ) Calculating I ( X ; Y ) is easy Channels with memory Assuming information stability [Dobrushin 1973] 1 C = lim n →∞ max nI ( X ; Y ) P X Calculating I ( X ; Y ) may be difficult For many interesting channels P XY has sparse support: -deletion, insertion, trapdoor,... Want lower bounds on I ( X ; Y ) that are useful for such channels Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | AEP: (¯ X , ¯ � � �� I ( X ; Y ) ≈ − log Pr Y ) ∈ T Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | AEP: (¯ X , ¯ � � �� I ( X ; Y ) ≈ − log Pr Y ) ∈ T (¯ X , ¯ � � Computing Pr Y ) ∈ T may be difficult Lower bound by replacing T with some S ⊇ T Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | ✘ ✘✘ AEP: (¯ X , ¯ � � �� I ( X ; Y ) ≥ − log Pr Y ) ∈ S (¯ X , ¯ � � Computing Pr Y ) ∈ T may be difficult Lower bound by replacing T with some S ⊇ T Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | ✘ ✘✘ AEP: � � I ( X ; Y ) ≥ − log E X E Y 1 { ( X , Y ) ∈S} (¯ X , ¯ � � Computing Pr Y ) ∈ T may be difficult Lower bound by replacing T with some S ⊇ T Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation X , Y are random vectors with joint distribution P XY X ∼ P X , ¯ ¯ Y ∼ P Y , ¯ ¯ X = Y | ✘ ✘✘ AEP: � � I ( X ; Y ) ≥ − log E X E Y 1 { ( X , Y ) ∈S} (¯ X , ¯ � � Computing Pr Y ) ∈ T may be difficult Lower bound by replacing T with some S ⊇ T A simple choice is the support S � { ( x , y ) : P XY ( x , y ) > 0 } Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation � � I ( X ; Y ) ≥ − log E X E Y 1 { ( X , Y ) ∈S} Bound relatively easy to compute: involves only marginals and support Gives reasonable results for certain “sparse” distributions Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation � � I ( X ; Y ) ≥ − log E X E Y 1 { ( X , Y ) ∈S} Bound relatively easy to compute: involves only marginals and support Gives reasonable results for certain “sparse” distributions Can we find better bounds that only involve marginals and support? Yes - replace S with ¯ S � { x ∈ T X , y ∈ T Y : P XY ( x , y ) > 0 } � � I ( X ; Y ) ≥ − log E X |T X E Y |T Y 1 { ( X , Y ) ∈S} [ Diggavi & Grossglauser ’01 ] [ Drinea & Mitzenmacher ’07 ] . . . Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Motivation � � I ( X ; Y ) ≥ − log E X E Y 1 { ( X , Y ) ∈S} Bound relatively easy to compute: involves only marginals and support Gives reasonable results for certain “sparse” distributions Can we find better bounds that only involve marginals and support? Yes - replace S with ¯ S � { x ∈ T X , y ∈ T Y : P XY ( x , y ) > 0 } � � I ( X ; Y ) ≥ − log E X |T X E Y |T Y 1 { ( X , Y ) ∈S} [ Diggavi & Grossglauser ’01 ] [ Drinea & Mitzenmacher ’07 ] . . . Main Result 1 { ( X , Y ) ∈S} I ( X ; Y ) ≥ − E Y log E X 1 { ( X , Y ) ∈S} − E X log E Y E X 1 { ( X , Y ) ∈S} Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Examples 1 { ( X , Y ) ∈S} I ( X ; Y ) ≥ − E Y log E X 1 { ( X , Y ) ∈S} − E X log E Y E X 1 { ( X , Y ) ∈S} Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Examples 1 { ( X , Y ) ∈S} I ( X ; Y ) ≥ − E Y log E X 1 { ( X , Y ) ∈S} − E X log E Y E X 1 { ( X , Y ) ∈S} When is this bound useful? Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Examples 1 { ( X , Y ) ∈S} I ( X ; Y ) ≥ − E Y log E X 1 { ( X , Y ) ∈S} − E X log E Y E X 1 { ( X , Y ) ∈S} When is this bound useful? Not for “fully-connected” channels: All pairs ( x , y ) ∈ S - the bound gives I ( X ; Y ) ≥ 0 Can be pretty good for channels with “low-connectivity” Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Example: Z-Channel 1 0 0 1 / 2 1 / 2 1 1 Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Example: Z-Channel 1 0 0 1 / 2 1 / 2 1 1 Bounds for IID Ber ( p ) Input � 1 � Mutual information: I ( X ; Y ) = H 2 (1 + p ) − (1 − p ) Naive bound: = − 1 1 − p � � � � I ( X , Y ) ≥ log 2 log 2 (1 − p ) E X E Y 1 { ( X , Y ) ∈S} Our bound: I ( X , Y ) ≥ − 1 � 1 � � 1 � 2(1 − p ) log(1 − p ) − p log 2(1 + p ) − (1 − p ) log 2(2 + p ) Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Example: Z-Channel 0.35 0.3 0.25 Bounds on I(X;Y) 0.2 0.15 0.1 Naive Lower Bound Lower Bound − 1st term 0.05 Lower Bound − both terms Mutual Information 0 0 0.2 0.4 0.6 0.8 1 p Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Preliminaries Channels via Conditional Probability Channel ⇔ Conditional distribution P Y | X Input alphabet X n Output alphabet Y ∗ Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Preliminaries Channels via Conditional Probability Channel ⇔ Conditional distribution P Y | X Input alphabet X n Output alphabet Y ∗ Channels via Actions (Functional Representation Lemma) P A - a distribution over mappings X n �→ Y ∗ Channel ⇔ Action A ∼ P A , A = X | Y = A ( X ) The choice of P A is not unique Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Preliminaries The Intrinsic Uncertainty Input distribution P X H ( A | X , Y ) is the intrinsic uncertainty Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Preliminaries The Intrinsic Uncertainty Input distribution P X H ( A | X , Y ) is the intrinsic uncertainty Capacity I ( X ; Y ) = H ( Y ) − H ( Y | X ) = H ( Y ) − ( H ( Y , A | X ) − H ( A | X , Y )) = H ( Y ) − H ( A | X ) − H ( Y | A , X ) + H ( A | X , Y ) = H ( Y ) − H ( A ) + H ( A | X , Y ) Lower bounding the intrinsic uncertainty = lower bounding MI Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Examples for Action Sets and Intrinsic Uncertainty The Binary Symmetric Channel Action ⇔ IID Noise sequence W ∼ Ber ( p ) Y = A ( X ) = X ⊕ W H ( A | X , Y ) = 0 Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Examples for Action Sets and Intrinsic Uncertainty The Binary Symmetric Channel Action ⇔ IID Noise sequence W ∼ Ber ( p ) Y = A ( X ) = X ⊕ W H ( A | X , Y ) = 0 The Z-Channel IID Noise sequence W ∼ Ber ( 1 Action ⇔ 2 ) � X i X i = 0 Y i = { A ( X ) } i = X i ⊕ W i X i = 1 Action masked when X i = 0 ⇒ H ( A | X , Y ) > 0 Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels
Recommend
More recommend
