INFORMATION THEORY FUNDAMENTALS AND MULTIPLE USER APPLICATIONS Max - PowerPoint PPT Presentation

Transmission over Unreliable Channels  The Channel Coding Problem: X Y 𝑋 W Channel Channel Channel 𝑞(𝑧|𝑦) Decoder Encoder  W  {1,2,…, 2 𝑜𝑆 } = message set of rate R  X = (x 1 x 2 … x n ) = codeword input to channel  Y = (y 1 y 2 … y n ) = codeword output from channel = decoded message P(error) = P{ W  𝑋}  𝑋

Shannon’s Channel Coding Theorem  Using the channel n times: X n • Y n • • • • • • • • •

Simple examples  Noiseless typewriter: 1 1 2 2 Input X Output Y 3 3 4 4 Number of noise free symbols = 4 Can transmit R = 𝑚𝑝𝑕 2 4 = 2 bits/transmission

Simple examples  Noisy typewriter (type 1): 0.5 1 1 0.5 2 2 Input X Output Y 0.5 3 3 0.5 4 4 Number of noise free symbols = 2 Can transmit R = 𝑚𝑝𝑕 2 2 = 1 bit/transmission

Simple examples  Noisy typewriter (type 2): 0.5 1 1 0.5 2 2 Input X Output Y 0.5 3 3 0.5 4 4 Number of noise free symbols = 2 (apparently, surprise later) Can transmit R ≥ 𝑚𝑝𝑕 2 2 = 1 bit/transmission

Simple examples  Noisy typewriter (type 3): 0.5 1 1 0.5 0.5 2 2 Input X Output Y 0.5 0.5 3 3 0.5 0.5 4 4 0.5 Number of noise free symbols = 2 Use X=1 and X=3 Can transmit R = 𝑚𝑝𝑕 2 2 = 1 bit/transmission

Simple examples  A tricky typewriter: 0.5 1 1 0.5 2 2 Input X Output Y 0.5 3 3 0.5 4 4 0.5 5 5 How many noise free symbols? Clearly at least 2, hopefully more.

Simple examples  Consider the n=2 extension of the channel: X 1 1 3 2 4 5 X 2 1 2 3 Which code 4 squares to pick? 5

Simple examples  Consider the n=2 extension of the channel: X 1 3 1 2 4 5 X 2 Let {X 1 ,X 2 } be 1 {(1,1), (2,3), 2 (3,5), (4,2), (5,4)} 3 4 5

Reminder of the channel  A tricky typewriter: 0.5 1 1 0.5 2 2 Input X Output Y 0.5 3 3 0.5 4 4 0.5 5 5 How many noise free symbols? Clearly at least 2, hopefully more.

Simple examples  Looking at the outputs: Y 1 3 1 2 4 5 Y 2 Let {X 1 ,X 2 } be 1 {(1,1), (2,3), 2 (3,5), (4,2), (5,4)} 3 4 5

Simple examples - observations  Note that we get 5 noise-free symbols in n=2 transmissions. 𝑚𝑝𝑕 2 5  Thus achieve rate 2 = 1.16 bits/transmission  with P(error) = 0.  For arbitrarily small P(error) can use long codes (n   ) to achieve log 2 5/2=1.32 bits/transmission, the channel capacity.

The Binary Symmetric Channel (BSC)  How many noise-free symbols? 1-  0 0  X Y  1-  1 1 A.: Clearly for n=1 there are none. How about using n large?

Shannon’s Second Theorem  Using the channel 𝑜 times: X n Y n • • • • •

Shannon’s Second Theorem  The Information Channel Capacity of a discrete memoryless channel is  𝐷 = max 𝐽(𝑌; 𝑍) . 𝑞(𝑦)  Note: 𝐽 𝑌; 𝑍 is a function of 𝑞 𝑦, 𝑧 = 𝑞 𝑦 𝑞 𝑧 𝑦 .  But 𝑞 𝑧 𝑦 is fixed by the channel.

Shannon’s Second Theorem  Direct Part:  If R < C  There exists a code with P(error)  0  Converse Part:  If R > C  Communication with P(error)  0  is not possible.

Shannon’s Second Theorem  Theorem: For a discrete memoryless channel, all rates 𝑆 below the information channel capacity 𝐷 are achievable with maximum probability of error arbitrarily small. Conversely, if the rate is above 𝐷, the probability of error is bounded away from zero.  Proof: Achievability: Use random coding to generate codes with a particular 𝑞 𝑦 distribution in the codewords. Then show that the average P(error) tends to zero with n   if 𝑆 < 𝐷. Then expurgate bad codewords to get a code with small maximum P(error) .

Shannon’s Second Theorem  Proof of Converse (sketch using AEP): Y n  2 𝑜𝐼(𝑍) Y n X n • • • • • typical balls  2 𝑜𝐼(𝑍|𝑌)

Shannon’s Second Theorem  Proof of Converse (sketch using AEP):  Recall the sphere packing problem. Maximum number of non-overlapping balls is bounded by 2 𝑜𝐼(𝑍) 2 𝑆 ≤ 2 𝑜𝐼(𝑍|𝑌) = 2 𝑜𝐽(𝑌:𝑍) ≤ 2 𝑜𝐷  Thus 2 𝑆 ≤ 2 𝐷 and 𝑆 ≤ 𝐷.  A formal proof uses Fano’s inequality.

Example: The Binary Symmetric Channel  1-  0 0  X Y  1-  1 1 1  C = max (H(Y) – H(Y|X)) C(  )  = 1 – h(  ) bits/transmission  Note: C=0 for  = ½ .  0 ½ 1

Example: The Binary Erasure Channel 1-   0 0  X E Y  1-  1 1 C(  )  C = max (H(Y) – H(Y|X))  = 1 –  bits/transmission Note: C=0 for  = 1.  Capacity is achieved with ½ 0 1 𝑞 𝑌 = 0 = 𝑞 𝑌 = 1 = ½ .

Example: The Z Channel  0 0 ½ X Y ½ 1 1 𝑐𝑗𝑢  C = max (H(Y) – H(Y|X)) = 𝑚𝑝𝑕 2 5 − 2 = 0.322 𝑢𝑠. 𝑞(𝑦) 2  Note: Maximizing 𝑞 𝑌 = 1 = 5 .  Homework: Obtain this capacity.

Changing Z channel into BEC  Show that code {01, 10} can transform a Z channel into a BEC.  What is a lower bound to capacity of the Z channel?

Typewriter type 2: ½  ½ ½ ½ Sum channel: 2 C = 2 C1 + 2 C2 where C 1 = C 2 = 0.322 C = 1,322 bits/channel use How many noise free symbols?

Example: Noisy typewriter ½  A A ½ B B C C X Y D D • E • • Z Z  C = max (H(Y) – H(Y|X))  = 𝑚𝑝𝑕 2 26 − 𝑚𝑝𝑕 2 2 = 𝑚𝑝𝑕 2 13 bits/transmission  Achieved with uniform distribution on the inputs.

Remark:  For this example, we can also achieve  𝐷 = 𝑚𝑝𝑕 2 13 bits/transmission with P(error)=0 and  n = 1 by transmitting alternating input symbols, i.e.,  X = {A C E … Z}.

Differential Entropy  Let 𝑌 be a continuous random variable with density 𝑔 𝑦 and support 𝑇 . The differential entropy of 𝑌 is ℎ 𝑌 = − 𝑔 𝑦 log 𝑔 𝑦 𝑒𝑦 (if it exists). 𝑇 Note: Also written as ℎ 𝑔 .

Examples: Uniform distribution  Let 𝑌 be uniform in the interval 0, 𝑏 . Then 1  𝑔 𝑦 = 𝑏 in the interval and 𝑔 𝑦 = 0 outside. 𝑏 1 1  ℎ 𝑌 = − 𝑏 𝑚𝑝𝑕 𝑏 𝑒𝑦 = log 𝑏 0  Note that ℎ 𝑌 can be negative for 𝑏 < 1.  However, 2 ℎ(𝑔) = 2 log 𝑏 = 𝑏 is the size of the support set, which is non-negative.

Example: Gaussian distribution −𝑦 2 1  Let 𝑌 ~  𝑦 = 2  2 𝑓𝑦𝑞 ( 2  2 ) 𝑦 2 2  2 − 𝑚𝑜 2  2 ] 𝑒𝑦  Then ℎ 𝑌 = ℎ  = −  𝑦 [− 𝐹𝑌 2 1 2 𝑚𝑜 2   2  = 2  2 + 1 2 𝑚𝑜 ( 2  e  2 ) nats  = 1 2 𝑚𝑝𝑕 ( 2  e  2 ) bits  Changing the base we have ℎ 𝑌 =

Relation of Differential and Discrete Entropies  Consider a quantization of X, denoted by X    Let X  = 𝑦 𝑗 i nside the 𝑗 th interval. Then 𝐼(𝑌  ) = - 𝑞 𝑗 𝑚𝑝𝑕 𝑞 𝑗 𝑗 = -  𝑔(𝑦 𝑗 ) 𝑚𝑝𝑕 𝑔(𝑦 𝑗 ) - 𝑚𝑝𝑕  𝑗  ℎ 𝑔 − log 

Differential Entropy  So the two entropies differ by the log of the quantization level  .  We can define joint differential entropy, conditional differential entropy, K-L divergence and mutual information with some care to avoid infinite differential entropies.

K-L divergence and Mutual Information  𝑔  𝐸(𝑔 g) = 𝑔 𝑚𝑝𝑕 𝑕 𝑔(𝑦,𝑧)  𝐽 𝑌; 𝑍 = 𝑔 𝑦, 𝑧 𝑚𝑝𝑕 𝑔 𝑦 𝑔(𝑧) 𝑒𝑦 𝑒𝑧  Thus , I(X;Y) = h(X) + h(Y) – h(X,Y). Note: h(X) can be negative, but I(X;Y) is always  0.

Differential entropy of a Gaussian vector  Theorem: Let 𝒀 be a Gaussian n -dimensional vector with mean  and covariance matrix 𝐿. Then 2 log((2  𝑓) 𝑜 𝐿 ) 1  ℎ 𝒀 =  where 𝐿 denotes the determinant of 𝐿.  Proof: Algebraic manipulation.

The Gaussian Channel Z~N (0, N I) X 𝑋 W Y Channel Channel + Decoder Encoder Power Constraint: EX 2 ≤P

The Gaussian Channel  Z~N (0, N I) X Y 𝑋 W Channel Channel + Decoder Encoder Power constraint: EX 2 ≤P  W  {1,2,…, 2 𝑜𝑆 } = message set of rate R  X = (x 1 x 2 … x n ) = codeword input to channel  Y = (y 1 y 2 … y n ) = codeword output from channel = decoded message P(error) = P{ W  𝑋}  𝑋

The Gaussian Channel  Using the channel n times: X n • Y n • • • • • • • • •

The Gaussian Channel   C𝑏𝑞𝑏𝑑𝑗𝑢𝑧 𝐷 = max 𝐽(𝑌; 𝑍) f(x): EX 2 ≤P  𝐽 𝑌; 𝑍 = ℎ 𝑍 − ℎ 𝑍 𝑌 = ℎ 𝑍 − ℎ 𝑌 + 𝑎|𝑌 1 1  = ℎ 𝑍 − ℎ 𝑎 ≤ 2 log 2  e 𝑄 + 𝑂 − 2 log 2  e 𝑂 1 𝑄  = 2 log 1 + 𝑂 bits/transmission

The Gaussian Channel   The capacity of the discrete time additive Gaussian channel: 1 𝑄  𝐷 = 2 log 1 + 𝑂 bits/transmission  achieved with X ~ N(0 , P).

Bandlimited Gaussian Channel  Consider the channel with continuous waveform inputs x(t) 𝑈 1 𝑈 𝑦 2 𝑢 𝑒𝑢 ≤ 𝑄) and Bandwidth with power constraint ( 0 limited to W. The channel has white Gaussian noise with power spectral density N 0 /2 watt/Hz.  In the interval (0,T) we can specify the code waveform by 2WT samples (Nyquist criterion). We can transmit these samples over discrete time Gaussian channels with noise variance N 0 /2. This gives 𝑄  𝐷 = 𝑋 log ( 1+ 𝑂 0 𝑋 ) bit/second

Bandlimited Gaussian Channel  𝑄  𝐷 = 𝑋 log ( 1+ 𝑂 0 𝑋 ) bit/second  Note: If W   𝑄 𝑂 0 𝑚𝑝𝑕 2 𝑓 bits/second.  we have C =

Bandlimited Gaussian Channel 𝑆 𝑋 be the spectral density  in bits per second  Let per Hertz. Also let 𝑄 = 𝐹 𝑐 𝑆 where 𝐹 𝑐 is the available energy per information bit.  We get 𝑆 𝐷 𝐹 𝑐 𝑆  𝑋 ≤ 𝑋 = log ( 1+ 𝑂 0 𝑋 ) bit/second.  Thus 2  −1 𝐹 𝑐  𝑂 0 ≥  This relation defines the so called Shannon Bound.

The Shannon Bound 2  −1 𝐹 𝑐  𝑂 0 ≥  𝐹 𝑐 𝐹 𝑐   𝑂 0 (dB) – 𝑂 0 Shannon Bound  0 0.69 -1.59 – 5 0.1 0.718 -1.44 – • 4 0.25 0.757 -1.21 0.5 0.828 -0.82 – 3 1 1 0 – • 2 2 1.5 1.76 𝐹 𝑐 – • 𝑂 0 (dB) 1 4 3.75 5.74  8 31.87 15.03        • -1 0 1 2 3 4 5 6

Shannon’s Water Filling Solution

Parallel Gaussian Channels  3 2.5 2 1

Example of Water Filling  Channels with noise levels 2, 1 and 3.  Available power = 2 1 0.5 1 1.5 1 0  Capacity= 2 log (1+ 2 ) + 2 log (1+ 1 ) + 2 log (1+ 3 )  Level of noise + signal power = 2.5  No power allocated to the third channel.

Parallel Gaussian Channels  3 2.5 2 1

Differential capacity Discrete memoryless channel as a band limited channel

Multiplex strategies (TDMA, FDMA)  P j  j

Multiplex strategies (non-orthonal CDMA) P j  1 𝑄 2 log (1 + 𝑂+ 𝑘−1 𝑄 ) j Discrete memoryless channel as a band limited channel M 𝑘 = 1 (1 + 𝑁𝑄 𝐷 2 log 𝑂 ) Aggregate capacity: : j=1

TDMA or FDMA versus CDMA Orthogonal schemes:  Bandwidth limitation (2WT dimensions) Number of Users Non-orthogonal CDMA (log has no cap) Aggregate Power

Multiple User Information Theory  Building Blocks:  Multiple Access Channels (MACs)  Broadcast Channels (BCs)  Interference Channels (IFCs)  Relay Channels (RCs)  Note: These channels have their discrete memoryless and Gaussian versions. For simplicity we will look at the Gaussian models.

Multiple Access Channel (MAC)

Gaussian Broadcast Channel

Superposition coding (1-  )P N 2  P 1

Superposition coding (1-  )P N 2  P 1

Standard Gaussian Interference Channel Power P1 ^ W 1 W 1 a b ^ W 2 W 2 Power P2

Symmetric Gaussian Interference Channel Power P Power P

Z-Gaussian Interference Channel

Interference Channel: Strategies Things that we can do with interference: Ignore (take interference as noise (IAN) 1. Avoid (divide the signal space (TDM/FDM)) 2. Partially decode both interfering signals 3. Partially decode one, fully decode the other 4. Fully decode both (only good for strong inter- 5. ference , a≥1)

Interference Channel: Brief history  Carleial (1975): Very strong interference does not reduce capacity (a 2 ≥ 1+P)  Sato (1981), Han and kobayashi (1981): Strong interference (a 2 ≥ 1) : IFC behaves like 2 MACs  Motahari, Khandani (2007), Shang, Kramer and Chen (2007), Annapureddy, Veeravalli (2007): Very weak interference (2a(1+a 2 P) ≤ 1 ) : Treat interference as noise – (IAN)

Interference Ch.: History (continued)  Sason (2004): Symmetrical superposition to beat TDM  Etkin, Tse, Wang (2008): capacity to within 1 bit  C (2011): Noisebergs to get Gaussian H+K for Z IFCs  C, Nair (2012, 2013, 2016, 2017): Some progress on achievable region of symmetric Gaussian IFCs  Polyanskiy and Wu, 2016: Corner points established.