Lecture 9 Polar Coding I-Hsiang Wang Department of Electrical - PowerPoint PPT Presentation

Polarization Lecture 9 Polar Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 29, 2015 1 / 25 I-Hsiang Wang IT Lecture 9

Polarization In Pursuit of Shannon’s Limit Since 1948, Shannon’s theory has drawn the sharp boundary between the possible and the impossible in data compression and data transmission. Once fundamental limits are characterized, the next natural question is: How to achieve these limits with acceptable complexity? For source coding , soon after Shannon’s 1948 paper, information and coding theorists found optimal compression schemes with low complexity: Huffman Code (1952): optimal for memoryless source Lempel-Ziv (1977): optimal for stationary ergodic source On the other hand, for channel coding , it turns out be a much harder problem. It has been the holy grail for coding theorist to find a coding scheme that achieves Shannon’s limit with low complexity. 2 / 25 I-Hsiang Wang IT Lecture 9

Polarization These codes perform very well empirically, but still in lack of theoretical I-Hsiang Wang 3 / 25 capacity (Shrinivas Kudekar, Tom Richardson, and Rüediger Urbanke). Later in 2012, spatially coupled LDPC codes were also shown to achieve The first provably capacity-achieving coding scheme with acceptable investigation on the performances and even proof of optimality. including turbo code, low-density parity-check (LDPC) code, etc. In Pursuit of Capacity-Achieving Codes Since 1990’s, there are several practical codes found to approach capacity, structures are hard to prove to achieve capacity. theorems, complexity issues are often neglected, while codes with 2 Lack of structure to reduce complexity. In the proof of coding proved that there exists coding schemes that achieve capacity. 1 Lack of explicit construction. In Shannon’s proof, it is only Two barriers in pursuing a low-complexity capacity-achieving codes: IT Lecture 9 complexity is polar code, introduced by Erdal Arıkan in 2007.

Polarization The paper wins the 2010 Information Theory Society Best Paper Award. 4 / 25 I-Hsiang Wang IT Lecture 9 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 7, JULY 2009 3051 Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels Erdal Arıkan , Senior Member, IEEE

Polarization Overview When Arıkan introduced polar codes in 2007, he focus on achieving capacity for the general binary-input memoryless symmetric channels (BMS), including BSC, BEC, etc. Later, polar codes are shown to be optimal in many other settings, including lossy source coding, non-binary-input channels, multiple access channels, source coding with side information (Wyner-Ziv problem), etc. Instead of giving a comprehensive introduction, we shall introduce channel polarization and polar coding for BMS, in the following order: 1 First we introduce the concept of channel polarization. 2 Then we explore polar coding. 5 / 25 I-Hsiang Wang IT Lecture 9

Polarization Beside, since we focus on BMS channels, and it is not difficult to prove I-Hsiang Wang 6 / 25 . achieves the channel capacity of any BMS, we shall use Notations IT Lecture 9 throughout this lecture: P to denote the input distribution p X Since the channel is the main focus, we shall use the following notations blocklength of the coding scheme. Recall in channel coding, we use the DMC N times with N being the W to denote the channel p Y | X I ( P , W ) to denote I ( X ; Y ) . ( 1 ) that X ∼ Ber 2 ( 1 ) I ( W ) (abuse of notation) to denote I ( P , W ) when the input P is Ber 2 In other words, the channel capacity of the BMS channel W is I ( W ) .

Polarization Basic Channel Transformation Channel Polarization 1 Polarization Basic Channel Transformation Channel Polarization 7 / 25 I-Hsiang Wang IT Lecture 9

Polarization N Usage of Channel W I-Hsiang Wang 8 / 25 Basic Channel Transformation IT Lecture 9 Channel Polarization Single Usage of Channel W X W Y X 1 Y 1 W X 2 Y 2 W ˆ ENC DEC M M . . . X N Y N W

Polarization Basic Channel Transformation I-Hsiang Wang 9 / 25 Apply special transforms to both input and output IT Lecture 9 Arıkan’s Idea Channel Polarization X 1 Y 1 V 1 U 1 W X 2 Y 2 U 2 W V 2 Pre- Post- Processing Processing . . . X N Y N U N W V N

Polarization Basic Channel Transformation I-Hsiang Wang 10 / 25 IT Lecture 9 Channel Polarization Arıkan’s Idea U 1 W 1 V 1 U 2 W 2 V 2 . . . U N W N V N

Polarization Basic Channel Transformation I-Hsiang Wang 11 / 25 IT Lecture 9 Arıkan’s Idea Channel Polarization Roughly N I ( W ) channels with capacity ≈ 1 U 1 W 1 V 1 U 2 W 2 V 2 . . . U N W N V N

Polarization Basic Channel Transformation I-Hsiang Wang 12 / 25 uncoded transmission, and throw those useless channels away. Coding becomes extremely simple: simply use those perfect channels for IT Lecture 9 Channel Polarization Arıkan’s Idea Roughly N I ( W ) channels with capacity ≈ 1 U 1 W 1 V 1 U 2 W 2 V 2 . . . Roughly N (1 − I ( W )) channels with capacity ≈ 0 U N W N V N Equivalently some perfect channels and some useless channels − → Polarization

Polarization Basic Channel Transformation Channel Polarization 1 Polarization Basic Channel Transformation Channel Polarization 13 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization Arıkan’s Basic Channel Transformation Consider two channel uses of W: 14 / 25 I-Hsiang Wang IT Lecture 9 X 1 W Y 1 X 2 W Y 2

Polarization Basic Channel Transformation Channel Polarization Arıkan’s Basic Channel Transformation Consider two channel uses of W: Apply the pre-processor: IT Lecture 9 I-Hsiang Wang 15 / 25 (Conservation of Information) (Polarization) The above transform yields the following two crucial phenomenon: . We now have two synthetic channels induced by the above procedure: U 1 W Y 1 X 1 = U 1 ⊕ U 2 , X 2 = U 2 , ( 1 U 2 W Y 2 ) where U 1 ⊥ ⊥ U 2 , U 1 , U 2 ∼ Ber 2 W − : U 1 → V 1 ≜ ( Y 1 , Y 2 ) W + : U 2 → V 2 ≜ ( Y 1 , Y 2 , U 1 ) I ( W − ) ≤ I ( W ) ≤ I ( W + ) I ( W − ) + I ( W + ) = 2 I ( W )

Polarization Basic Channel Transformation I-Hsiang Wang 16 / 25 = = Example 1 Example: Binary Erasure Channel Channel Polarization IT Lecture 9 Let W be a BEC with erasure probability ε ∈ (0 , 1) , and I ( W ) = 1 − ε . Find the values of I ( W − ) and I ( W + ) , and verify the above properties. sol : Intuitively W − is worse than W and W + is better than W: For W − , input is U 1 , output is ( Y 1 , Y 2 ) . Only when both Y 1 and Y 2 are not erased, one can figure out U 1 ! ⇒ W − is BEC with erasure probability 1 − (1 − ε ) 2 = 2 ε − ε 2 . For W + , input is U 2 , output is ( Y 1 , Y 2 , U 1 ) . As long as one of Y 1 and Y 2 are not erased, one can figure out U 2 ! ⇒ W + is BEC with erasure probability ε 2 . Hence, I ( W − ) = 1 − 2 ε + ε 2 and I ( W + ) = 1 − ε 2 .

Polarization Basic Channel Transformation Channel Polarization Example: Binary Symmetric Channel Example 2 17 / 25 I-Hsiang Wang IT Lecture 9 Let W be a BSC with crossover probability p ∈ (0 , 1) , and I ( W ) = 1 − H b ( p ) . Find the values of I ( W − ) and I ( W + ) .

Polarization pf : We prove the conservation of information first: I-Hsiang Wang 18 / 25 property holds. (Proof of the condition for equality is left as exercise.) Basic Channel Transformation I IT Lecture 9 Theorem 1 Arıkan’s basic transformation, we have Basic Properties Channel Polarization For any BMS channel W and the induced channels { W − , W + } from I ( W − ) ≤ I ( W ) ≤ I ( W + ) with equality iff I ( W ) = 0 or 1 . I ( W − ) + I ( W + ) = 2 I ( W ) W − ) W + ) ( ( + I = I ( U 1 ; Y 1 , Y 2 ) + I ( U 2 ; Y 1 , Y 2 , U 1 ) = I ( U 1 ; Y 1 , Y 2 ) + I ( U 2 ; Y 1 , Y 2 | U 1 ) = I ( U 1 , U 2 ; Y 1 , Y 2 ) = I ( X 1 , X 2 ; Y 1 , Y 2 ) = I ( X 1 ; Y 1 ) + I ( X 2 ; Y 2 ) = 2 I ( W ) . I ( W + ) = I ( X 2 ; Y 1 , Y 2 , U 1 ) ≥ I ( X 2 ; Y 2 ) = I ( W ) , and hence the first

Polarization Basic Channel Transformation I-Hsiang Wang 19 / 25 Upper boundary: Lower boundary: BSC minimizes the stretch BEC maximizes the stretch that among all BMS channels: If we plot the “information stretch” (Taken from Chap. 12.1 of Moser[4] .) IT Lecture 9 Extremal Channels Channel Polarization BEC I ( W + ) − I ( W − ) versus the original 0.5 information I ( W ) , it can be shown I ( W + ) − I ( W − ) [bits] 0.4 0.3 BSC 0.2 0.1 2 H b (2 p (1 − p )) − 2 H b ( p ) , 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 − 1 (1 − I ( W )) . I ( W ) [bits] where p = H b 2 I ( W ) (1 − I ( W )) .

Polarization Basic Channel Transformation Channel Polarization 1 Polarization Basic Channel Transformation Channel Polarization 20 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization Recursive Application of Arıkan’s Transformation 21 / 25 I-Hsiang Wang IT Lecture 9 Duplicate W, apply the transformation, and get W − and W + . W W

Polarization Basic Channel Transformation Channel Polarization Recursive Application of Arıkan’s Transformation 22 / 25 I-Hsiang Wang IT Lecture 9 Duplicate W, apply the transformation, and get W − and W + . Duplicate W − (and W + ). W W W W