Chapter 1: Introduction to Information Theory Book: Information - PowerPoint PPT Presentation

Chapter 1: Introduction to Information Theory Book: “Information Theory, Inference, and Learning Algorithms” from David MacKay

Noisy Channels - Error correcting Codes Examples - System solution - Channel models - Binary symmetric Channel • Modem → phone line → modem • Parental cell → DNA → daugh- ter cells • ESA → radio waves in space → Beagle 2 • RAM → hdd → RAM How to reduce the probability of error?

Noisy Channels - Error correcting Codes Examples - System solution - Channel models - Binary symmetric Channel ˆ s t Noisy channel r s Received ✲ Decoder ✲ ✲ ✲ Source Encoder p ( r | t ) Message Source coding (Data Compression) Channel Error Correction Channel Decompression Source Entropy Coding Channel Capacity Deconding • Data Compression What transmission rates are acievable (Removing Redundancy) with infinitely small error. → Source Coding Theorem: What compression rates are achievable. • Encryption: Between Source- and Channel Coding • Error Correction (Adding redundancy) → Channel Coding Theorem: • Decoding & Encoding should be fast.

Noisy Channels - Error correcting Codes Examples - System solution - Channel models - Binary symmetric Channel • Gaussian channel: • Binary symmetric channel: (1 − f ) z i ✲ 0 0 ❅ � ✒ � ❅ � ❄ ❅ ✓✏ f � ❅ ✲ � x i y i ❅ ✒✑ � ❅ 2 πσ 2 exp � � � ❘ ❅ − ( y − x ) 2 1 P ( y | x ) = √ ✲ 2 σ 2 1 1 (1 − f ) P ( y = 0 | x = 0) = 1 − f P ( y = 0 | x = 1) = f • Noiseless binary channel: P ( y = 1 | x = 0) = f P ( y = 0 | x = 0) = 1 − f ✲ 0 0 • Noisy typewriter channel ✲ PPPPPPPP A A f f ✲ P q PPPPPPPP B B ✲ 1 1 f f ✲ q P PPPPPPPP C C P ( y = 0 | x = 0) = P ( y = 1 | x = 1) = 1 f f ✲ P q . . . P ( y = A | x = A ) = 1 − f P ( y = B | x = A ) = f P ( y = B | x = B ) = 1 − f P ( y = C | x = B ) = f · · · · · ·

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity Coding Theory • The object of coding is to in- or corrupted, it will still be pos- troduce redundancy so that if sible to recover the message at some of the information is lost the receiver.

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity Repetition Codes (e.g. R 3 : 0 → 000 and 1 → 111 ) (1 − f ) ✲ s 0 0 0 0 1 0 1 1 0 ❅ � ✒ ���� ���� ���� ���� ���� ���� ���� � t 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 ❅ f � ❅ � n 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 ❅ ❅ ❘ � ✲ r 0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 0 1 1 (1 − f ) • Optimal decoding? • Odds ratio: P ( s =1 | r ) P ( s =0 | r ) = P ( r | s =1) P ( r | s =0) = � 3 P ( r n | t n (1)) n =1 P ( r n | t n (0)) Most probable p ( s | r ) • assume: p (0) = p (1) = 1 2 • For a single bit • P ( s | r 1 r 2 r 3 ) = P ( r 1 r 2 r 3 | s ) P ( s ) • bin. sym. channel P ( r 1 r 2 r 3 ) • if P ( s = 1 | r ) > P ( s = 0 | r ) decode  (1 − f ) : r n = 1  ˆ s = 1 else ˆ s = 0 f P ( r n | t n (1)) • P ( r n | t n (0)) = � � − 1 BSC: P ( r | s ) = P ( r | t ( s )) = � 3 (1 − f ) n =1 P ( r n | t n ( s )) : r n = 0  f

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity � ≫ 1 � Likelihood ratio P ( r | s = 1 ) γ = 1 − f Received sequence r Decoded sequence ˆ s P ( r | s = 0 ) f γ − 3 000 0 γ − 1 001 0 γ − 1 010 0 γ − 1 100 0 γ 1 101 1 γ 1 110 1 γ 1 011 1 γ 3 111 1 s 0 0 1 0 1 1 0 ���� ���� ���� ���� ���� ���� ���� t 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 n 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 r 0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 0 0 0 ���� ���� ���� ���� ���� ���� ���� ˆ s 0 0 1 0 0 1 0 corrected errors ⋆ undetected errors ⋆

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity What do we gain by using R 3 ? Two possibilities for errors, which follow the binomial distribution: � N � f e (1 − f ) N − e p ( e | f, N ) = r • All three bits flipped p #3 = f 3 • Just two bits flipped p #2 = 3 f 2 (1 − f ) Probability of error in R 3 is p B = p b = f 3 + 3 f 2 (1 − f ) = 3 f 2 − 2 f 3

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity Error rate of R N Codes • Error when at least ⌈ N/ 2 ⌉ bits in one block are flipped. N � N � � f n (1 − f ) N − n p B = n n =( N +1) / 2 • For small f this term is dominated by n = ( N +1) . 2 • Def.: The (transmission) rate R = log( M ) • Concatenated codes: R 2 bits per transmission. 3 = R 3 ◦ R 3 N 3 f 2 � 2 = 27 f 4 � p b ( R 2 3 ) ≈ 3 � 9 � f 5 (1 − f ) 4 ≈ 126 f 5 p b ( R 9 ) ≈ 5 • The rate of R 3 is R = 1 3 . but R 2 3 requires less computation

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity • Parity check code: • linear code: t = Gs (7 , 4) − Hamming Code :   1 0 0 0 information bits parity bits 0 1 0 0     0 0 1 0 � �   I 4 � �� � � �� � ssssssssssssssssss ssssssssss   • G = = 0 0 0 1   P s 1 s 2 s 3 s 4 t 5 t 6 t 7  1 1 1 0    0 1 1 1   1 0 1 1 s t s t s t s t 0000 0000000 0100 0100110 1000 1000101 1100 1100011 0001 0001011 0101 0101101 1001 1001110 1101 1101000 0010 0010111 0110 0110001 1010 1010010 1110 1110100 0011 0011100 0111 0111010 1011 1011001 1111 1111111

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity • Parity check code: • linear code: t = Gs (7 , 4) − Hamming Code :   1 0 0 0 information bits parity bits 0 1 0 0     0 0 1 0 � �   I 4 � �� � � �� � ssssssssssssssssss ssssssssss   • G = = 0 0 0 1   P s 1 s 2 s 3 s 4 t 5 t 6 t 7  1 1 1 0    0 1 1 1   1 0 1 1 s t s t s t s t 0000 0000000 0100 0100110 1000 1000101 1100 1100011 0001 0001011 0101 0101101 1001 1001110 1101 1101000 0010 0010111 0110 0110001 1010 1010010 1110 1110100 0011 0011100 0111 0111010 1011 1011001 1111 1111111

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity • Parity check code: • linear code: t = Gs (7 , 4) − Hamming Code :   1 0 0 0 information bits parity bits 0 1 0 0     0 0 1 0 � �   I 4 � �� � � �� � ssssssssssssssssss ssssssssss   • G = = 0 0 0 1   P s 1 s 2 s 3 s 4 t 5 t 6 t 7  1 1 1 0    0 1 1 1   1 0 1 1 s t s t s t s t 0000 0000000 0100 0100110 1000 1000101 1100 1100011 0001 0001011 0101 0101101 1001 1001110 1101 1101000 0010 0010111 0110 0110001 1010 1010010 1110 1110100 0011 0011100 0111 0111010 1011 1011001 1111 1111111

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity • Parity check code: • linear code: t = Gs (7 , 4) − Hamming Code :   1 0 0 0 information bits parity bits 0 1 0 0     0 0 1 0 � �   I 4 � �� � � �� � ssssssssssssssssss ssssssssss   • G = = 0 0 0 1   P s 1 s 2 s 3 s 4 t 5 t 6 t 7  1 1 1 0    0 1 1 1   1 0 1 1 s t s t s t s t 0000 0000000 0100 0100110 1000 1000101 1100 1100011 0001 0001011 0101 0101101 1001 1001110 1101 1101000 0010 0010111 0110 0110001 1010 1010010 1110 1110100 0011 0011100 0111 0111010 1011 1011001 1111 1111111

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity Decoding scheme • Minimal distance between code • parity-check matrix: words is 3 � 1 � 1 1 0 1 0 0 � � 0 1 1 1 0 1 0 H = P I 3 = 1 0 1 1 0 0 1 • For the binary symmetric chan- nel and equiprobable source • ∀ code words: vectors s One deconding   0 scheme is to take the “closest”   Ht = 0   vector 0 min d ( r , t ( s )) s = ⇒ Search all possible sources. • syndrome vector: z = Hr

Noisy Channels - Error correcting Codes Repetition Codes - Block Codes - Channel capacity Correcting Errors • Example: • Pictoral solution: Transmit s = 1000 Encoded t = 1000101 Noise n = 0100000 Received r = 1100101 � � T • z = Hr = 1 1 0 Syndrome z 000 001 010 011 100 101 110 111 Unflip this bit r 7 r 6 r 4 r 5 r 1 r 2 r 3 none