Chapter 10 Rate Distortion Theory Peng-Hua Wang Graduate Inst. of - PowerPoint PPT Presentation

Chapter 10 Rate Distortion Theory Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei University

Chapter Outline Chap. 10 Rate Distortion Theory 10.1 Quantization 10.2 Definitions 10.3 Calculation of the Rate Distortion Function 10.4 Converse to the Rate Distortion Theorem 10.5 Achievability of the Rate Distortion Function 10.6 Strongly Typical Sequences and Rate Distortion 10.7 Characterization of the Rate Distortion Function 10.8 Computation of Channel Capacity and the Rate Distortion Function Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 2/22

10.1 Quantization Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 3/22

Introduction ■ Finite representation of a continuous r.v. ◆ can’t be perfect ◆ How well can we do ? ⇒ need to define “goodness” or distortion measurement means the distance between a r.v. and its representation ◆ This is in fact the lossy compression Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 4/22

Quantization ■ Let X be a r.v., ˆ X = ˆ X ( X ) be its representation. ■ If we quantize X into R bits, means we use 2 R distinct values to represent X . ■ Problem. Find optimal set ˆ X , called the representation points or code points, and the region associated with each value in ˆ X such that certain error measurement is minimized. Example. X ∼ N (0 , σ 2 ) , R = 1 , min E [( X − ˆ X ) 2 ] .  � 2 X ≥ 0 π σ,  ˆ X = � 2 − X < 0 π σ,  Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 5/22

Quantization Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 6/22

10.2 Definitions Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 7/22

Definitions Definition 1 (Distortion Function) d : X × ˆ X → R + x ∈ ˆ x ) ≥ 0 for x ∈ X , ˆ X . which means a function d with d ( x, ˆ x ) < ∞ ■ A distortion function d ( x, ˆ x ) is bounded if max d ( x, ˆ ■ The distortion between sequences x and ˆ x is defined by n x n ) = 1 � d ( x n , ˆ d ( x i , ˆ x i ) n i =1 ■ Example. x ) = 1 if x � = ˆ ◆ Hamming distance. d ( x, ˆ x and d ( x, ˆ x ) = 0 if x = ˆ x. x ) 2 . ◆ squared-error distortion. d ( x, ˆ x ) = ( x − ˆ Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 8/22

Definitions Definition 2 (Rate Distortion Codes) A 2 nR , n -rate distortion code consists of an encoding function f n : X n → { 1 , 2 , . . . , 2 nR } and a decoding function g n : { 1 , 2 , . . . , 2 nR } → X n . The distortion D associated with the code is the average distortion over all codewords � D = E [ d ( X n , g n ( f n ( X n )))] = p ( x n ) d ( x n , g n ( f n ( x n ))) x n We may call ˆ X n the vector quantization, reproduction, reconstruction, source code, or estimation of X. Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 9/22

Definitions Definition 3 (Achievable) A rate distortion pair ( R, D ) is said to be achievable if there exists a sequence of (2 nR , n ) -rate distortion code ( f n , g n ) with n →∞ E [ d ( X n , g n ( f n ( X n )))] ≤ D. lim ■ A rate distortion region for a source is the closure of the set of achievable rate distortion pairs ( R, D ) . ■ The rate distortion function R ( D ) for a source is the infimum of rates R such that ( R, D ) is in the rate distortion region of the source for a given distortion D . ■ The rate distortion function R ( D ) for a source is the infimum of all distortion D such that ( R, D ) is in the rate distortion region of the source for a given rate R . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 10/22

Definitions Definition 4 (Information Rate Distortion Function) The information rate distortion function R ( I ) ( D ) for a source X with distortion measure d ( x, ˆ x ) is defined as x ) ≤ D I ( X ; ˆ R ( I ) ( D ) = min X ) p (ˆ x | x ): � x p ( x ) p (ˆ x | x ) d ( x, ˆ x, ˆ Theorem 1 (Rate Distortion Function) The rate distortion function for an i.i.d. source X with distribution p ( x ) a nd bounded distortion function d ( x, ˆ x ) is equal to the associated information rate distortion function. Thus, R ( D ) = R ( I ) ( D ) is the minimum achievable rate at distortion D . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 11/22

10.3 Calculation of the Rate Distortion Function Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 12/22

Binary Source Theorem 2 The rate distortion function for a Bernoulli ( p ) source with Hamming distortion is given by � H ( p ) − H ( D ) , 0 ≤ D ≤ min { p, 1 − p } R ( D ) = D > min { p, 1 − p } 0 , ■ If D ≥ p , we can achieve R ( D ) = 0 (one code to represent two values) by letting ˆ X = 0 since the distortion is x = 0) × d ( x = 0 , ˆ p ( x = 1 , ˆ x = 1) = p ( x = 1) p (ˆ x = 0 | x = 1) × 1 = p � �� � =1 Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 13/22

Binary Source Proof. Let Z = 0 if X = ˆ X and Z = 1 if X � = ˆ X. Denote p ( Z = 1) = t . The distortion E [ d ( X, ˆ X ] = p ( X = 0 , ˆ X = 1) × 1 + p ( X = 1 , ˆ X = 0) × 1 = p ( Z = 1) = t ≤ D I ( X ; ˆ X ) = H ( X ) − H ( X | ˆ X ) = H ( p ) − H ( Z | ˆ X ) ≥ H ( p ) − H ( Z ) = H ( p ) − H ( t ) ≥ H ( p ) − H ( D ) Equality holds when H ( X | ˆ X ) = H ( D ) . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 14/22

Binary Source Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 15/22

Gaussian Source Theorem 3 The rate distortion function for a N (0 , σ 2 ) source with squared error distortion is � 2 log σ 2 1 0 ≤ D ≤ σ 2 D , R ( D ) = D > σ 2 0 , ■ If D ≥ σ 2 , we can achieve R ( D ) = 0 (one code to represent ALL values) by letting ˆ X = 0 since the distortion is � � x ) 2 φ ( x ) dx = x 2 φ ( x ) dx = σ 2 ( x − ˆ where φ ( x ) is the pdf of N (0 , σ 2 ) . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 16/22

Gaussian Source Proof. I ( X ; ˆ X ) = h ( X ) − h ( X | ˆ X ) = h ( X ) − h ( X − ˆ X | ˆ X ) ≥ h ( X ) − h ( X − ˆ X ) ≥ h ( X ) − h ( N (0 , E [( X − ˆ X ) 2 ])) = 1 2 log(2 πeσ 2 ) − 1 2 log(2 πeE [( X − ˆ X ) 2 ]) ≥ 1 2 log(2 πeσ 2 ) − 1 2 log(2 πeD ) 2 log σ 2 = 1 D Equality holds when Z = X − ˆ X has a normal distribution of zero mean and variance D . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 17/22

Gaussian Source Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 18/22

Sphere Packing for Channel Coding Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 19/22

Sphere Packing for Channel Coding For each sent codeword, the received codeword is contained in a √ nN . The received vectors have energy no grater sphere of radius � than n ( P + N ) , so they lie in a sphere of radius n ( P + N ) . How many codeword can we use without intersection in the decoding sphere? � n �� n ( P + N ) A n � n/ 2 � 1 + P √ M = = N nN ) n A n ( where A the constant for calculating the volume of n -dimensional sphere. For example, A 2 = π , A 3 = 4 3 π. Therefore, the capacity is � � 1 n log M = 1 1 + P 2 log . N Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 20/22

Sphere Packing for Rate Distortion Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 21/22

Sphere Packing for Rate Distortion Consider a Gaussian source of variance σ 2 . A (2 nR , n ) rate distortion code for this source with distortion D is a set of M = 2 nR sequences √ in R n . All these sequences lie within a sphere of radius nσ 2 . The √ D of some codewords. How source sequences are within a distance many codeword can we use without intersection in the decoding sphere? � √ � n A n nσ 2 � n/ 2 � σ 2 √ M = = D A n ( nD ) n Therefore, the rate is 2 log σ 2 n log M = 1 1 D . Peng-Hua Wang, May 21, 2012 Information Theory, Chap. 10 - p. 22/22