FUNDAMENTALS AND MULTIPLE USER APPLICATIONS (PART II) Max H. M. - PowerPoint PPT Presentation

Changing Z channel into BEC  Show that code {01, 10} can transform a Z channel into a BEC.  What is a lower bound to capacity of the Z channel?

Typewriter type 2: ½  ½ ½ ½ Sum channel: 2 C = 2 C1 + 2 C2 where C 1 = C 2 = 0.322 C = 1,322 bits/channel use How many noise free symbols?

Example: Noisy typewriter ½  A A ½ B B C C X Y D D • E • • Z Z  C = max (H(Y) – H(Y|X))  = 𝑚𝑝𝑕 2 26 − 𝑚𝑝𝑕 2 2 = 𝑚𝑝𝑕 2 13 bits/transmission  Achieved with uniform distribution on the inputs.

Remark:  For this example, we can also achieve  𝐷 = 𝑚𝑝𝑕 2 13 bits/transmission with P(error)=0 and  n = 1 by transmitting alternating input symbols, i.e.,  X = {A C E … Z}.

Differential Entropy  Let 𝑌 be a continuous random variable with density 𝑔 𝑦 and support 𝑇 . The differential entropy of 𝑌 is ℎ 𝑌 = − 𝑔 𝑦 log 𝑔 𝑦 𝑒𝑦 (if it exists). 𝑇 Note: Also written as ℎ 𝑔 .

Examples: Uniform distribution  Let 𝑌 be uniform in the interval 0, 𝑏 . Then 1  𝑔 𝑦 = 𝑏 in the interval and 𝑔 𝑦 = 0 outside. 𝑏 1 1  ℎ 𝑌 = − 𝑏 𝑚𝑝𝑕 𝑏 𝑒𝑦 = log 𝑏 0  Note that ℎ 𝑌 can be negative for 𝑏 < 1.  However, 2 ℎ(𝑔) = 2 log 𝑏 = 𝑏 is the size of the support set, which is non-negative.

Example: Gaussian distribution −𝑦 2 1  Let 𝑌 ~  𝑦 = 2  2 𝑓𝑦𝑞 ( 2  2 ) 𝑦 2 2  2 − 𝑚𝑜 2  2 ] 𝑒𝑦  Then ℎ 𝑌 = ℎ  = −  𝑦 [− 𝐹𝑌 2 1 2 𝑚𝑜 2   2  = 2  2 + 1 2 𝑚𝑜 ( 2  e  2 ) nats  = 1 2 𝑚𝑝𝑕 ( 2  e  2 ) bits  Changing the base we have ℎ 𝑌 =

Relation of Differential and Discrete Entropies  Consider a quantization of X, denoted by X    Let X  = 𝑦 𝑗 i nside the 𝑗 th interval. Then 𝐼(𝑌  ) = - 𝑞 𝑗 𝑚𝑝𝑕 𝑞 𝑗 𝑗 = -  𝑔(𝑦 𝑗 ) 𝑚𝑝𝑕 𝑔(𝑦 𝑗 ) - 𝑚𝑝𝑕  𝑗  ℎ 𝑔 − log 

Differential Entropy  So the two entropies differ by the log of the quantization level  .  We can define joint differential entropy, conditional differential entropy, K-L divergence and mutual information with some care to avoid infinite differential entropies.

K-L divergence and Mutual Information  𝑔  𝐸(𝑔 g) = 𝑔 𝑚𝑝𝑕 𝑕 𝑔(𝑦,𝑧)  𝐽 𝑌; 𝑍 = 𝑔 𝑦, 𝑧 𝑚𝑝𝑕 𝑔 𝑦 𝑔(𝑧) 𝑒𝑦 𝑒𝑧  Thus , I(X;Y) = h(X) + h(Y) – h(X,Y). Note: h(X) can be negative, but I(X;Y) is always  0.

Differential entropy of a Gaussian vector  Theorem: Let 𝒀 be a Gaussian n -dimensional vector with mean  and covariance matrix 𝐿. Then 2 log((2  𝑓) 𝑜 𝐿 ) 1  ℎ 𝒀 =  where 𝐿 denotes the determinant of 𝐿.  Proof: Algebraic manipulation.

The Gaussian Channel Z~N (0, N I) X 𝑋 W Y Channel Channel + Decoder Encoder Power Constraint: EX 2 ≤P

The Gaussian Channel  Z~N (0, N I) X Y 𝑋 W Channel Channel + Decoder Encoder Power constraint: EX 2 ≤P  W  {1,2,…, 2 𝑜𝑆 } = message set of rate R  X = (x 1 x 2 … x n ) = codeword input to channel  Y = (y 1 y 2 … y n ) = codeword output from channel = decoded message P(error) = P{ W  𝑋}  𝑋

The Gaussian Channel  Using the channel n times: X n • Y n • • • • • • • • •

The Gaussian Channel   C𝑏𝑞𝑏𝑑𝑗𝑢𝑧 𝐷 = max 𝐽(𝑌; 𝑍) f(x): EX 2 ≤P  𝐽 𝑌; 𝑍 = ℎ 𝑍 − ℎ 𝑍 𝑌 = ℎ 𝑍 − ℎ 𝑌 + 𝑎|𝑌 1 1  = ℎ 𝑍 − ℎ 𝑎 ≤ 2 log 2  e 𝑄 + 𝑂 − 2 log 2  e 𝑂 1 𝑄  = 2 log 1 + 𝑂 bits/transmission

The Gaussian Channel   The capacity of the discrete time additive Gaussian channel: 1 𝑄  𝐷 = 2 log 1 + 𝑂 bits/transmission  achieved with X ~ N(0 , P).

Bandlimited Gaussian Channel  Consider the channel with continuous waveform inputs x(t) 𝑈 1 𝑈 𝑦 2 𝑢 𝑒𝑢 ≤ 𝑄) and Bandwidth with power constraint ( 0 limited to W. The channel has white Gaussian noise with power spectral density N 0 /2 watt/Hz.  In the interval (0,T) we can specify the code waveform by 2WT samples (Nyquist criterion). We can transmit these samples over discrete time Gaussian channels with noise variance N 0 /2. This gives 𝑄  𝐷 = 𝑋 log ( 1+ 𝑂 0 𝑋 ) bit/second

Bandlimited Gaussian Channel  𝑄  𝐷 = 𝑋 log ( 1+ 𝑂 0 𝑋 ) bit/second  Note: If W   𝑄 𝑂 0 𝑚𝑝𝑕 2 𝑓 bits/second.  we have C =

Bandlimited Gaussian Channel 𝑆 𝑋 be the spectral density  in bits per second  Let per Hertz. Also let 𝑄 = 𝐹 𝑐 𝑆 where 𝐹 𝑐 is the available energy per information bit.  We get 𝑆 𝐷 𝐹 𝑐 𝑆  𝑋 ≤ 𝑋 = log ( 1+ 𝑂 0 𝑋 ) bit/second.  Thus 2  −1 𝐹 𝑐  𝑂 0 ≥  This relation defines the so called Shannon Bound.

The Shannon Bound 2  −1 𝐹 𝑐  𝑂 0 ≥  𝐹 𝑐 𝐹 𝑐   𝑂 0 (dB) – 𝑂 0 Shannon Bound  0 0.69 -1.59 – 5 0.1 0.718 -1.44 – • 4 0.25 0.757 -1.21 0.5 0.828 -0.82 – 3 1 1 0 – • 2 2 1.5 1.76 𝐹 𝑐 – • 𝑂 0 (dB) 1 4 3.75 5.74  8 31.87 15.03        • -1 0 1 2 3 4 5 6

Shannon’s Water Filling Solution

Parallel Gaussian Channels  3 2.5 2 1

Example of Water Filling  Channels with noise levels 2, 1 and 3.  Available power = 2 1 0.5 1 1.5 1 0  Capacity= 2 log (1+ 2 ) + 2 log (1+ 1 ) + 2 log (1+ 3 )  Level of noise + signal power = 2.5  No power allocated to the third channel.

Parallel Gaussian Channels  3 2.5 2 1

Differential capacity Discrete memoryless channel as a band limited channel

Multiplex strategies (TDMA, FDMA)  P j  j 𝑘 = 1 (1 + 𝑄 𝐷 2 log Aggregate capacity: : 𝑂 )

Multiplex strategies (non-orthonal CDMA) P j  1 𝑄 2 log (1 + 𝑂+ 𝑘−1 𝑄 ) j Discrete memoryless channel as a band limited channel M 𝑘 = 1 (1 + 𝑁𝑄 𝐷 2 log 𝑂 ) Aggregate capacity: : j=1

TDMA or FDMA versus CDMA Orthogonal schemes:  Bandwidth limitation (2WT dimensions) Number of Users Non-orthogonal CDMA (log has no cap) Aggregate Power

Multiple User Information Theory  Building Blocks:  Multiple Access Channels (MACs)  Broadcast Channels (BCs)  Interference Channels (IFCs)  Relay Channels (RCs)  Note: These channels have their discrete memoryless and Gaussian versions. For simplicity we will look at the Gaussian models.

Multiple Access Channel  A well understood model.  Models the uplink channel in wireless comm. X 1 W 1 Encoder ^ ^ W 1 ,W 2 Y P(y|x 1 ,x 2 ) Decoder X 2 Encoder W 2 Capacity region obtained by Ahlswede (1971) and Liao (1972)

Capacity region - MAC  C = closure of convex hull of { (R1,R2) s.t.  R 1 ≤ I(X 1 ; Y | X 2 ),  R 2 ≤ I(X 2 ; Y | X 1 ),  R 1 + R 2 ≤ I(X 1 , X 2 ; Y ) for all p 1 (x 1 ) . p 2 (x 2 ) }

Multiple Access Channel (MAC)

Broadcast Channel (Cover, 1972)  Still open in discrete memoryless case.  Models the downlink channel in wireless comm. Y 1 ^ W 1 Decoder X W 1 ,W 2 Encoder P(y 1 ,y 2 |x) Y 2 ^ W 2 Decoder Cover introduced superposition coding.

Superposition coding  Message Clouds Y 1 can see the message, Y 2 can see the cloud center

Gaussian Broadcast Channel

Superposition coding (1-  )P N 2  P 1

Superposition coding (1-  )P N 2  P 1

Interference Channel  Gaussian Interference Channel - standard form  Brief history  Z-Interference channel  Symmetric Interference channel

Standard Gaussian Interference Channel Power P1 ^ W 1 W 1 a b ^ W 2 W 2 Power P2

Symmetric Gaussian Interference Channel Power P Power P

Z-Gaussian Interference Channel

Interference Channel: Strategies Things that we can do with interference: Ignore (take interference as noise (IAN) 1. Avoid (divide the signal space (TDM/FDM)) 2. Partially decode both interfering signals 3. Partially decode one, fully decode the other 4. Fully decode both (only good for strong inter- 5. ference , a≥1)

Interference Channel: Brief history  Carleial (1975): Very strong interference does not reduce capacity (a 2 ≥ 1+P)  Sato (1981), Han and kobayashi (1981): Strong interference (a 2 ≥ 1) : IFC behaves like 2 MACs  Motahari, Khandani (2007), Shang, Kramer and Chen (2007), Annapureddy, Veeravalli (2007): Very weak interference (2a(1+a 2 P) ≤ 1 ) : Treat interference as noise – (IAN)

Interference Ch.: History (continued)  Sason (2004): Symmetrical superposition to beat TDM  Etkin, Tse, Wang (2008): capacity to within 1 bit  Polyanskiy and Wu, 2016: Corner points established.

Summary: Z interference Channels  Z-Gaussian Interference Channel as a degraded interference channel  Discrete Memoryless Channel as a band limited channel  Multiplex Region: growing Noisebergs  Overflow Region: back to superposition

Z-Gaussian Interference Channel

Degraded Gaussian Interference Channel

Differential capacity Discrete memoryless channel as a band limited channel

Interference x Broadcast Channels

Superposition coding (1-  )P N 2  P 1

Superposition coding (1-  )P N 2  P 1

Degraded Interference Channel - One Extreme Point

Degraded Interference Channel - Another Extreme Point

Intermediary Points (Multiplex Region)

Admissible region for (  , h)

Intermediary Point (Overflow Region)

Admissible region h Q 1 =1 Q 2 = 1 a = 0.5 N 2 = 3 

The Z-Gaussian Interference Channel Rate Region R 2 Q 1 =1 Q 2 = 1 a = 0.5 N 2 = 3 R 1

Admissible region h Q 1 =1 Q 2 = 1 a = 0.99 N 2 = 0.02 

The Z-Gaussian Interference Channel Rate Region R 2 Q 1 =1 Q 2 = 1 a = 0.99 N 2 = 0.02 R 1