Lecture 4 Channel Coding I-Hsiang Wang Department of Electrical - PowerPoint PPT Presentation

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation Summary Lecture 4 Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 29, 2014 1 / 53 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Meta Description I-Hsiang Wang 2 / 53 certain decoding criterion. w from the channel output y N . . determining the stochastic relationship between the output symbol p Achievability Proof and Source-Channel Separation and a family of conditional distributions NIT Lecture 4 Summary The Channel Coding Problem x N y N Channel Noisy Channel b w w Encoder Channel Decoder 1 Message : Random message W ∼ Unif [1 : 2 K ] . 2 Channel : Consist of an input alphabet X , an output alphabet Y , { ( y k | x k , y k − 1 ) } | k ∈ N ( x k − 1 , y k − 1 ) y k and the input symbol x k along with all past signals 3 Encoder : Encode the message w by a length N codeword x N ∈ X N . 4 Decoder : Reconstruct message � 5 Efficiency : Maximize the code rate R := K N bits/channel use, given

Channel Capacity and the Weak Converse e I-Hsiang Wang 3 / 53 Note: In lossless source coding, we see that the infimum of compression code rate R where vanishing error probability is possible? e Is it possible to have a sequence of encoder/decoder pairs such attention to answering the following question: Following the development of lossless source coding, Shannon turned the Ans : Probably not, unless the channel noise has some special structure. Question : Is it possible to get zero error probability? . W Achievability Proof and Source-Channel Separation NIT Lecture 4 Summary Decoding Criterion: Vanishing Error Probability x N y N Channel Noisy Channel b w w Encoder Channel Decoder { } A key performance measure: Error Probability P ( N ) W ̸ = � := Pr that P ( N ) → 0 as N → ∞ ? If so, what is the largest possible rates where vanishing error probability is possible is H ( { S i } ) .

Channel Capacity and the Weak Converse e I-Hsiang Wang 4 / 53 finite length V e error exponent e Achievability Proof and Source-Channel Separation capacity NIT Lecture 4 Summary Rate R Block Probability Length of Error P ( N ) N e Take N → ∞ , Require P ( N ) → 0 : sup R = C . Take N → ∞ : min P ( N ) ≈ 2 − NE ( R ) . √ n Q − 1 ( ϵ ) . Fix N , Require P ( N ) ≤ ϵ : sup R ≈ C −

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation Summary In this course we only focus on capacity. In other words, we ignore the issue of delay and do not pursue finer analysis of the error probability via large deviation techniques. 5 / 53 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse is memoryless if I-Hsiang Wang 6 / 53 the channel output y N ? , Question : is our definition of a channel sufficient to specify p transition function. . p Achievability Proof and Source-Channel Separation NIT Lecture 4 p A discrete channel Summary Discrete Memoryless Channel (DMC) In order to demonstrate the key ideas in channel coding, in this lecture we shall focus on discrete memoryless channels (DMC) defined below. Definition 1 (Discrete Memoryless Channel) ( { ( y k | x k , y k − 1 ) } ) X , | k ∈ N , Y ∀ k ∈ N , ( y k | x k , y k − 1 ) = p Y | X ( y k | x k ) . ( X k − 1 , Y k − 1 ) In other words, Y k − X k − Here the conditional p.m.f. p Y | X is called the channel law or channel ( y N | x N ) the stochastic relationship between the channel input (codeword) x N and

Channel Capacity and the Weak Converse N I-Hsiang Wang 7 / 53 past channel output, i.e., feedback . function, which implies that the encoder can potentially make use of the is induced by the encoding p Interpretation : . cannot be obtained from p , which p Hence, we need to further specify Achievability Proof and Source-Channel Separation p p p N Summary p p NIT Lecture 4 ( x N , y N ) ( y N | x N ) = p p ( x N ) ∏ ( x N , y N ) ( x k , y k | x k − 1 , y k − 1 ) = k =1 ∏ ( y k | x k , y k − 1 ) ( x k | x k − 1 , y k − 1 ) = k =1 { ( x k | x k − 1 , y k − 1 ) } | k ∈ N ( x N ) { ( x k | x k − 1 , y k − 1 ) } | k ∈ N

Channel Capacity and the Weak Converse With Feedback I-Hsiang Wang 8 / 53 N without feedback, p For a DMC Proposition 1 (DMC without Feedback) . suffices to specify p p specifying Achievability Proof and Source-Channel Separation realization of the channel output, then, p Suppose in the system, the encoder has no knowledge about the NIT Lecture 4 Summary No Feedback DMC without Feedback x k x k y k y k Channel Noisy Channel Noisy w w Encoder Channel Encoder Channel y k − 1 D ( x k | x k − 1 , y k − 1 ) ( x k | x k − 1 ) = p for all k ∈ N , and it is said the the channel has no feedback. In this case, { ( y k | x k , y k − 1 ) } ( y N | x N ) | k ∈ N ∏ ( ) ( y N | x N ) X , p Y | X , Y = p Y | X ( y i | x i ) . k =1

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation I-Hsiang Wang 9 / 53 3 Prove the achievability part with a random coding argument. 2 Prove the converse part: an achievable rate cannot exceed C . couple of examples to show how to compute channel capacity. 1 Give the problem formulation, state the main theorem, and visit a To demonstrate this beautiful result, we organize this lecture as follows: The above holds regardless of the availability of feedback. NIT Lecture 4 For a DMC , the maximum code rate with described) noisy channel coding theorem due to Shannon: In this lecture, we would like to establish the following (informally Overview Summary ( ) X , p Y | X , Y vanishing error probability is the channel capacity C := max p X ( · ) I ( X ; Y ) .

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation Summary Channel Capacity Proof of the Weak Converse 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse 2 Achievability Proof and Source-Channel Separation Achievability Proof Source-Channel Separation 3 Summary 10 / 53 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse 1 A I-Hsiang Wang 11 / 53 e 3 A rate R is said to be achievable if there exist a sequence of . W e w or an Achievability Proof and Source-Channel Separation channel code consists of NIT Lecture 4 Proof of the Weak Converse Channel Coding without Feedback: Problem Setup Summary Channel Capacity x N y N Channel Noisy Channel b w w Encoder Channel Decoder ( ) 2 NR , N an encoding function (encoder) enc N : [1 : 2 K ] → X N that maps each message w to a length N codeword x N , where K := ⌈ NR ⌉ . a decoding function (decoder) dec N : Y N → [1 : 2 K ] ∪ {∗} that maps a channel output sequence y N to a reconstructed message � error message ∗ . { } 2 The error probability is defined as P ( N ) W ̸ = � := Pr ( ) codes such that P ( N ) 2 NR , N → 0 as N → ∞ . The channel capacity is defined as C := sup { R | R : achievable } .

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation I-Hsiang Wang 12 / 53 maximized. input distribution so that the amount of information transfer is from the channel output Y . amount of information about the channel input X that one can infer (1) NIT Lecture 4 following: Theorem 1 (Channel Coding Theorem for DMC without Feedback) Channel Coding Theorem for Discrete Memoryless Channel Proof of the Weak Converse Channel Capacity Summary The capacity of the DMC p ( y | x ) without feedback is given by the C = max p ( x ) I ( X ; Y ) . The capacity formula (1) is intuitive, since I ( X ; Y ) represents the The maximization over p ( x ) stands for choosing the best possible

Channel Capacity and the Weak Converse Achievability Proof and Source-Channel Separation I-Hsiang Wang 13 / 53 argument called random coding . encoding/decoding schemes such that the error probability vanishes vanishing error probability ( converse ). NIT Lecture 4 compute capacity. 1 First we give some examples of noisy channels to show how to In the following, we Proof of the Weak Converse Channel Capacity Summary 2 Then, we prove that for any rate R > C , it is impossible to have 3 Finally, we prove that for any R < C , there exist a sequence of as blocklength tends to ∞ ( achievability ), based on a probabilistic

Channel Capacity and the Weak Converse p I-Hsiang Wang 14 / 53 Ans : Yes, choose X to be uniform = Achievability Proof and Source-Channel Separation . p NIT Lecture 4 Channel Capacity A binary symmetric channel (BSC) consists of Summary Proof of the Weak Converse Binary Symmetric Channel Y X 1 − p 0 0 Binary input/output X = Y = { 0 , 1 } . p [ 1 − p ] Channel law p ( y | x ) = 1 − p p The capacity of BSC is C BSC = 1 − H b ( p ) . 1 1 1 − p To compute BSC capacity, observe I ( X ; Y ) = H ( Y ) − H ( Y | X ) , and H ( Y | X = 0) = H ( Y | X = 1) = H b ( p ) = ⇒ H ( Y | X ) = H b ( p ) . H ( Y ) ≤ log 2 = 1 , with equality iff Y is uniform. Question : Is it possible to choose a p ( x ) such that Y is uniform? ⇒ C = max p ( x ) I ( X ; Y ) = 1 − H b ( p ) .