Information Theory Lecture 10 Network Information Theory (CT15); a - PDF document

Information Theory Lecture 10 • Network Information Theory (CT15); a focus on channel capacity results • The (two-user) multiple access channel (15.3) • The (two-user) broadcast channel (15.6) • The relay channel (15.7) • Some remarks on general multiterminal channels (15.10) Mikael Skoglund, Information Theory 1/25 Joint Typicality • Extension of previous results to an arbitrary number of variables (most basic defs here, many additional results in CT) • Notation • For any k -tuple x k 1 = ( x 1 , x 2 , . . . , x k ) ∈ X 1 × X 2 × · · · × X k and subset of indices S ⊆ { 1 , 2 , . . . , k } let x S = ( x i ) i ∈S • Assume x i ∈ X n i , any i , and let x S be a matrix with x i as rows for i ∈ S . Let the |S| -tuple x S ,j be the j th column of x S . • As in CT, a n . = 2 n ( c ± ε ) means � � 1 � � n log a n − c � < ε, � � � for all sufficiently large n Mikael Skoglund, Information Theory 2/25

• For random variables X k 1 with joint distribution p ( x k 1 ) : Generate X S via n independent copies of x S ,j , j = 1 , . . . , n . Then, n � p ( x S ,j ) � p ( x S ) Pr( X S = x S ) = j =1 • For S ⊆ { 1 , 2 , . . . , k } , define the set of ε -typical n -sequences x S : Pr( X S ′ = x S ′ ) . � = 2 − n [ H ( X S′ ) ± ε ] , ∀S ′ ⊆ S � A ( n ) ε ( S ) = • Then, for any ε > 0 , sufficiently large n , and S ⊆ { 1 , . . . , k } , P ( A ( n ) ε ( S )) ≥ 1 − ε p ( x S ) . = 2 − n [ H ( X S ) ± ε ] if x S ∈ A ( n ) ε ( S ) ε ( S ) | . | A ( n ) = 2 n [ H ( X S ) ± 2 ε ] Mikael Skoglund, Information Theory 3/25 The Multiple Access Channel encoder 1 channel X 1 α 1 ( · ) W 1 decoder ˆ W 1 Y p ( y | x 1 , x 2 ) β ( · ) ˆ W 2 encoder 2 X 2 W 2 α 2 ( · ) • Two “users” communicating over a common channel. (The generalization to more than two is straightforward.) Mikael Skoglund, Information Theory 4/25

Coding : • Memoryless pmf (or pdf): p ( y | x 1 , x 2 ) , x 1 ∈ X 1 , x 2 ∈ X 2 , y ∈ Y • Data: W 1 ∈ I 1 = { 1 , . . . , M 1 } and W 2 ∈ I 2 = { 1 , . . . , M 2 } • Assume W 1 and W 2 uniformly distributed and independent • Encoders: α 1 : I 1 → X n and α 2 : I 2 → X n 1 2 • Rates: R 1 = 1 n log M 1 and R 2 = 1 n log M 2 • Decoder: β : Y n → I 1 × I 2 , β ( Y n ) = ( ˆ W 1 , ˆ W 2 ) • Error probability: P ( n ) ( ˆ W 1 , ˆ � � = Pr W 2 ) � = ( W 1 , W 2 ) e Mikael Skoglund, Information Theory 5/25 Capacity : We have two (or more) rates, R 1 and R 2 = ⇒ cannot consider one maximum achievable rate = ⇒ study sets of achievable rate-pairs ( R 1 , R 2 ) = ⇒ trade-off between R 1 and R 2 • Achievable rate-pair: ( R 1 , R 2 ) is achievable if ( α 1 , α 2 , β ) exists such that P ( n ) → 0 as n → ∞ e • Capacity region : The closure of the set of all achievable rate-pairs ( R 1 , R 2 ) Mikael Skoglund, Information Theory 6/25

Capacity Region for the Multiple Access Channel • Fix π ( x 1 , x 2 ) = p 1 ( x 1 ) p 2 ( x 2 ) on X 1 and X 2 . � X n � � X n � Draw 1 ( i ) : i ∈ I 1 and 2 ( j ) : j ∈ I 2 in an i.i.d. manner according to p 1 and p 2 . • Symmetry of codebook generation = ⇒ P ( n ) ( ˆ W 1 , ˆ � � = Pr W 2 ) � = ( W 1 , W 2 ) e ( ˆ W 1 , ˆ � � � = Pr W 2 ) � = (1 , 1) � ( W 1 , W 2 ) = (1 , 1) where the second “Pr” is with respect to the channel and the random codebook design. Mikael Skoglund, Information Theory 7/25 • Also ( ˆ W 1 , ˆ = Pr( ˆ W 1 � = 1 , ˆ � � Pr W 2 ) � = (1 , 1) W 2 � = 1) + Pr( ˆ W 1 � = 1 , ˆ W 2 = 1) + Pr( ˆ W 1 = 1 , ˆ W 2 � = 1) = P ( n ) 12 + P ( n ) + P ( n ) 1 2 conditioned that ( W 1 , W 2 ) = (1 , 1) everywhere. • Joint typicality decoding, declare ( ˆ W 1 , ˆ W 2 ) = (1 , 1) if ∈ A ( n ) X n 1 ( i ) , X n 2 ( j ) , Y n � � ε only for i = j = 1 ⇒ P ( n ) 12 ≤ 2 n [ R 1 + R 2 − I ( X 1 ,X 2 ; Y )+4 ε ] P ( n ) ≤ 2 n [ R 1 − I ( X 1 ; Y | X 2 )+3 ε ] 1 P ( n ) ≤ 2 n [ R 2 − I ( X 2 ; Y | X 1 )+3 ε ] 2 Mikael Skoglund, Information Theory 8/25

R 2 A I ( X 2 ; Y | X 1 ) B I ( X 2 ; Y ) R 1 I ( X 1 ; Y ) I ( X 1 ; Y | X 2 ) • Hence, for a fixed π ( x 1 , x 2 ) = p 1 ( x 1 ) p 2 ( x 2 ) the capacity region contains at least all pairs ( R 1 , R 2 ) in the set Π defined by R 1 < I ( X 1 ; Y | X 2 ) R 2 < I ( X 2 ; Y | X 1 ) R 1 + R 2 < I ( X 1 , X 2 ; Y ) Mikael Skoglund, Information Theory 9/25 • The corner points • Consider the point ‘ A ’ � R 1 = I ( X 1 ; Y ) R 1 + R 2 = I ( X 1 , X 2 ; Y ) R 2 = I ( X 2 ; Y | X 1 ) • User 1 ignores the presence of user 2 ⇒ R 1 = I ( X 1 ; Y ) • Decode user 1’s codeword ⇒ User 2 sees an equivalent channel with input X n 2 and output ( Y n , X n 1 ) ⇒ R 2 = I ( X 2 ; Y, X 1 ) = I ( X 2 ; Y | X 1 ) + I ( X 1 ; X 2 ) = I ( X 2 ; Y | X 1 ) • The above can be repeated with 1 ↔ 2 and A ↔ B • Points on the line A – B can be achieved by time sharing Mikael Skoglund, Information Theory 10/25

• Each particular choice of distribution π gives an achievable region Π ; for two different π ’s, R 2 π 1 π 2 R 1 • Fixed π = ⇒ Π is convex. Varying π = ⇒ Π can be non-convex. However all rates on a line connecting two achievable rate-pairs are achievable by time-sharing. Mikael Skoglund, Information Theory 11/25 • The capacity region for the multiple access channel is the closure of the convex hull of the set of points defined by the three inequalities R 1 < I ( X 1 ; Y | X 2 ) R 2 < I ( X 2 ; Y | X 1 ) R 1 + R 2 < I ( X 1 , X 2 ; Y ) over all possible product distributions p 1 ( x 1 ) p 2 ( x 2 ) for ( X 1 , X 2 ) . • Proof : Achievability proof based on jointly typical sequences (as shown before) and a “time-sharing variable”. Converse proof based on Fano’s inequality and the independence of X n 1 and X n 2 (since they are functions of independent messages). Mikael Skoglund, Information Theory 12/25

Example: A Gaussian Channel • Bandlimited AWGN channel with two additive users Y ( t ) = X 1 ( t ) + X 2 ( t ) + Z ( t ) . The noise Z ( t ) is zero-mean Gaussian with power spectral density N 0 / 2 , and X 1 ( t ) and X 2 ( t ) are subject to the power constraints P 1 and P 2 , respectively. The available bandwidth is W . • The capacity of the corresponding single-user channel (with power constraint P ) is � P � W · C [bits/second] WN 0 where C ( x ) = log(1 + x ) . Mikael Skoglund, Information Theory 13/25 • Time-Division Multiple-Access (TDMA) : Let user 1 use all of the bandwidth with power P 1 /α a fraction α ∈ [0 , 1] of time, and let user 2 use all the bandwidth with power P 2 / (1 − α ) the remaining fraction 1 − α of time. The achievable rates then are � P 1 /α � � P 2 / (1 − α ) � R 1 < W · α C R 2 < W · (1 − α ) C WN 0 WN 0 • Frequency-Division Multiple-Access (FDMA) : Let user 1 transmit with power P 1 using a fraction α of the available bandwidth W , and let user two transmit with power P 2 the remaining fraction (1 − α ) W . The achievable rates are � P 1 � � P 1 � R 1 < αW · C R 2 < (1 − α ) W · C αWN 0 (1 − α ) WN 0 • TDMA and FDMA are equivalent from a capacity perspective! Mikael Skoglund, Information Theory 14/25

• Code-Division Multiple-Access (CDMA) : Defined, in our context, as all schemes that can be implemented to achieve the rates in the true capacity region � P 1 � � P 1 � R 1 ≤ W · C = W log 1 + WN 0 WN 0 � P 2 � � P 2 � R 2 ≤ W · C = W log 1 + WN 0 WN 0 � P 1 + P 2 � � � 1 + P 1 + P 2 R 1 + R 2 ≤ W · C = W log WN 0 WN 0 Mikael Skoglund, Information Theory 15/25 T/FDMA R 2 R 2 CDMA T/FDMA CDMA I I R 1 R 1 I 2 I Capacity region for P 1 = P 2 Capacity region for P 1 = 2 P 2 1 − α = P 1 α Note that T/FDMA is only optimal when . P 2 Mikael Skoglund, Information Theory 16/25

The Broadcast Channel channel decoder 1 ( ˆ W 0 , ˆ W 1 ) Y 1 β 1 ( · ) encoder ( W 0 , W 1 , W 2 ) X α ( · ) p ( y 1 , y 2 | x ) decoder 2 ( ˆ W 0 , ˆ W 2 ) Y 2 β 2 ( · ) • One transmitter, several receivers • Message W 0 is a public message for both receivers, whereas W 1 and W 2 are private messages Mikael Skoglund, Information Theory 17/25 The Degraded Broadcast Channel Y 1 Y 1 Y 2 X X ⇐ ⇒ p ( y 1 , y 2 | x ) p ( y 1 | x ) p ( y 2 | y 1 ) Y 2 • A broadcast channel is degraded if it can be split as in the figure. That is, Y 2 is a “noisier” version of X than Y 1 , p ( y 1 , y 2 | x ) = p ( y 2 | y 1 ) p ( y 1 | x ) . • The Gaussian and the binary symmetric broadcast channels are degraded (see the examples in CT). Mikael Skoglund, Information Theory 18/25