Today Proof of Converse Coding Theorem Intuition: For message m , - PDF document

Today Proof of Converse Coding Theorem • Intuition: For message m , let S m ⊆ { 0 , 1 } n • More on Shannon’s theory be the set of received words that decode to − Proof of converse. m. ( S m = D − 1 ( m ) ). − Few words on generality. − Contrast with Hamming theory. • Average size of D ( m ) = 2 n − k . • Back to error-correcting codes: Goals. • Volume of disc of radius pn around E ( m ) is 2 H ( p ) n . • Tools: • Intuition: If volume ≫ 2 n − k can’t have this − Probability theory: − Algebra: Finite fields, Linear spaces. ball decoding to m — but we need to! • Formalize? � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 1 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 2 Proof of Converse Coding Theorem (contd.) Prob. [correct decoding ] � η ∈ { 0 , 1 } n � = Pr[ m sent , η error and m ∈{ 0 , 1 } k 2 − k · 2 � � � ≤ Pr[ η error] + m η ∈ B ( p ′ n,n ) η �∈ B ( p ′ n,n ) exp( − n ) + 2 − k − H ( p ′ ) n · � ≤ I m,η m,η Let I m,η be the indicator variable that is 1 iff exp( − n ) + 2 − k − H ( p ′ ) n · 2 n D (( E ( m ) + η )) = m . = ≤ exp( − n ) Let p ′ < p be such that R > 1 − H ( p ′ ) . � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 3 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 4

Generalizations of Shannon’s theorem Some of the main contributions • Channels more general • Rigorous Definition of elusive concepts: Information, Randomness. − Input symbols Σ , Output symbols Γ , where both may be infinite • Mathematical tools: Entropy, Mutual (reals/complexes). information, Relative entropy. − Channel given by its probability transition matrix P = P σ,γ . • Theorems: Coding theorem, converse. − Channel need not be independent - could be Markovian (remembers finite amount • Emphasis on the “feasible” as opposed to of state in determining next error bit). “done”. • In almost all cases: random coding + mld works. • Always non-constructive. � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 5 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 6 Contrast between Hamming and • Most important difference: modelling of Shannon error — adversarial vs. probabilistic. Accounts for the huge difference in our ability to analyze one while having gaps in the other. • Works intertwined in time. • Nevertheless good to build Hamming like • Hamming’s work focusses on distance, and codes, even when trying to solve the image of E . Shannon problem. • Shannon’s work focusses on probabilities only (no mention of distance) and E, D but not properties of image of E . • Hamming’s results more constructive, definitions less so. • Shannon’s results not constructive, though definitions beg constructivitty. � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 7 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 8

Our focus • Will combine with analysis of encoding complexity and decoding complexity. • Codes, and associated encoding and decoding functions. • Distance is not the only measure, but we will say what we can about it. • Code parameters: n, k, d, q ; • typical goal: given three optimize fourth. • Coarser goal: consider only R = k/n , δ = d/n and q and given two, optimize the third. • In particular, can we get R, δ > 0 for constant q ? � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 9 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 10 Tools Finite fields and linear error-correcting codes • Probability tools: • Field: algebraic structure with addition, − Linearity of expections, Union bound. multiplication, both commutative and − Expectation of product of independent associative with inverses, and multiplication] r.v.s distributive over addition. − Tail inqualities: Markov, Chebychev, Chernoff. • Finite field: Number of elements finite. Well known fact: field exists iff size is a • Algebra prime power. See lecture notes on algebra − Finite fields. for further details. Denote field of size q by − Vector spaces over finite fields. F q . • Elementary combinatorics and algorithmics. • Vector spaces: V defined over a field F . Addition of vectors, multiplication of vector with “scalar” (i.e., field element) is defined, � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 11 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 12

Why study this category? and finally an inner product (product of two vectors yielding a scalar is defined). • If alphabet is a field, then ambient space • Linear codes are the most common. Σ n becomes a vector space F n q . • Seem to be as strong as general ones. • If a code forms a vector space within F n q then it is a linear code. Denoted [ n, k, d ] q • Have succinct specification, efficient code. encoding and efficient error-detecting algorithms. Why? (Generator matrix and Parity check matrix.) • Linear algebra provides other useful tools: Duals of codes provide interesting constructions. • Dual of linear code is code generated by transpose of parity check matrix. � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 13 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 14 Example: Dual of Hamming codes the identity matrix. Such matrices are called Hadamard matrices, and hence the code is called a Hadamard code.) • Message m = � m 1 , . . . , m ℓ � . • Moral of the story: Duals of good codes • Encoding given by �� m , x �� x ∈ F ℓ 2 − 0 . end up being good. No proven reason. • Fact: (will prove later): m � = 0 implies Pr x [ �� m , x � = 0] = 1 2 • Implies dual of [2 ℓ − 1 , 2 ℓ − ℓ − 1 , 3] 2 Hamming code is a [2 ℓ − 1 , ℓ, 2 ℓ − 1 ] code. • Often called the simplex code or the Hadamard code. (If we add a coordinate that is zero to all coordinates, and write 0 s as − 1 s, then the matrix whose rows are all the codewords form a +1 / − 1 matrix whose product with its transpose is a multiple of � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 15 � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 16

Next few lectures • Towards asymptotically good codes: − Some good codes that are not asymptotically good. − Some compositions that lead to good codes. � Madhu Sudan, Fall 2004: Essential Coding Theory: MIT 6.895 c 17