Structure of Optimal Quantizer for Binary-Input Continuous-Output - PowerPoint PPT Presentation

Structure of Optimal Quantizer for Binary-Input Continuous-Output Channels with Output Constraints Thuan Nguyen and Thinh Nguyen Oregon State University nguyeth9@oregonstate.edu, thinhq@oregonstate.edu June 6, 2020 Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 1 / 23

Outline Introduction 1 Problem Formulation 2 Structure of optimal quantizer 3 Applications 4 Conclusion 5 Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 2 / 23

1. Introduction Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 3 / 23

Quantization maximizing mutual information • Designing quantizer maximizing mutual information receives tremendous attention [1,2,3,4,5,6,7,8,9,10,11]. • Finding the optimal quantizer under some output constraints has a long history: entropy-constrained scalar quantization [12], [13], [14], entropy-constrained vector quantization [15], [16], [17]. → We investigate the problem of designing optimal quantizers that maximize the mutual information between input and the quantized output under quantized output constraints. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 4 / 23

Related work • Strouse et al. proposed an iterative algorithm to find a local optimal quantizer that maximizes the mutual information under the entropy constraint of output [18]. • Gyorgy and Linder proved that convex cell quantizers are optimal for entropy-constrained scalar quantization [14]. • Kurkoski and Yagi showed that if the input is binary then the optimal quantizer maximizing mutual information must belong to class of convex cell quantizers [1]. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 5 / 23

Our Results • The optimal quantizer maximizing mutual information under any output constraint must belong to the class of convex cell quantizers . • A fast algorithm for finding a globally optimal quantizer if the channel input is binary. • A sufficient condition for which a single threshold quantizer is optimal. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 6 / 23

2. Problem formulation Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 7 / 23

Problem formulation • Binary random input X = { x 1 , x 2 } , p X = [ p x 1 , p x 2 ] = [ p 1 , p 2 ] . • Continuous output Y is specified by two given conditional densities p y | x 1 = φ 1 ( y ) and p y | x 2 = φ 2 ( y ) . • Quantizer Q is used to quantize Y to a discrete output set Z = { z 1 , z 2 , . . . , z N } . Our objective is to find the solution to the following optimization problem: max Q β I ( X ; Z ) − C ( p Z ) , (1) Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 8 / 23

Notations p 1 φ 1 ( y ) • r y = p x 1 | y = p 1 φ 1 ( y ) + p 2 φ 2 ( y ) . • v y = p x | y = [ p x 1 | y , p x 2 | y ] = [ r y , 1 − r y ] . • µ ( y ) denotes the density function of Y , µ ( y ) = p 1 φ 1 ( y ) + p 2 φ 2 ( y ) . • Z i denotes the set of y that is quantized to the i th output z i . Z i = { y : Q ( y ) = z i } . Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 9 / 23

Definitions Definition ( Kullback-Leibler (KL) divergence ) KL divergence of two probability vectors a = ( a 1 , a 2 , . . . , a J ) and b = ( b 1 , b 2 , . . . , b J ) is defined by J a i log( a i � D ( a || b ) = ) . (2) b i i = 1 Definition ( Centroid ) Centroid of subset Z i ⊂ R is a two dimensional vector c i = [ c i , 1 − c i ] that globally minimizes the total KL divergence v y to c i from all y ∈ Z i : � c i = arg min D ( v y || c ) µ ( y ) dy . (3) y ∈ Z i c Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 10 / 23

Definitions cont. Definition ( Vector order ) For two given conditional probability vectors v y 1 and v y 2 , we define v y 1 ≤ v y 2 if and only if p x 1 | y 1 ≤ p x 1 | y 2 or r y 1 ≤ r y 2 . Similarly, for two centroid vectors c i = [ c i , 1 − c i ] and c j = [ c j , 1 − c j ] , c i ≤ c j if and only if c i ≤ c j . Definition ( Set order ) Given two arbitrary sets A ⊂ R and B ⊂ R , we define A ≤ B if and only if for all y a ∈ A and any y b ∈ B , we have v y a ≤ v y b . We define A ≡ B if and only if A ⊂ B and B ⊂ A . Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 11 / 23

Definitions cont. Definition ( Convex cell quantizer ) A quantizer is a convex cell quantizer in r y if Q ∗ ( r y ) = z i , if a ∗ i − 1 ≤ r y < a ∗ (4) i , for some optimal thresholds a ∗ 0 = 0 < a ∗ 1 < · · · < a ∗ N − 1 < a ∗ N = 1. In other words, Q is convex cell quantizer if Z 1 ≤ Z 2 ≤ · · · ≤ Z N . Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 12 / 23

3. Structure of optimal quantizer Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 13 / 23

Optimal quantizer and KL-divergence • A quantizer Q maximizing I ( X ; Z ) is equivalent to minimizing the average distortion of KL divergence E Y [ D KL ( v y || c i )] = D ( Q ) [2]. • If D ( Q ) ≤ D ( Q ′ ) then I ( X ; Z ) Q ≥ I ( X ; Z ) Q ′ . Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 14 / 23

Structure of optimal quantizer for binary partition ( N = 2) Theorem Let Q be an arbitrary quantizer that produces two disjoint output sets { Z 1 , Z 2 } corresponding to two centroid vectors c 1 , c 2 such that c 1 ≤ c 2 , there exists a convex cell quantizer ¯ Q with two output sets { ¯ Z 1 , ¯ Z 2 } and c 2 } such that ¯ Z 1 ≤ ¯ the corresponding centroids { ¯ c 1 , ¯ Z 2 , p Z i = p ¯ Z i for i = 1 , 2 and D ( ¯ Q ) ≤ D ( Q ) . Proof. For any given quantizer Q ( y ) , we show that existing a convex cell quantizer ¯ Q ( r y ) such that: • p Z = p ¯ Z . • D ( ¯ Q ) ≤ D ( Q ) . → The optimal quantizer must belong to the class of convex cell quantizers . Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 15 / 23

Optimal quantizer structure for N > 2 Theorem Let Q be a quantizer with arbitrary disjoint quantized-output sets { Z 1 , Z 2 , . . . , Z N } corresponding to N centroids c 1 , c 2 , . . . , c N such that c i ≤ c i + 1 ∀ i , there exists an other convex cell quantizer ¯ Q with the output sets { ¯ Z 1 , ¯ Z 2 , . . . , ¯ Z N } and the corresponding centroids { ¯ c 1 , ¯ c 2 , . . . , ¯ c N } such that ¯ Z i ≤ ¯ Z i ∀ i and D ( ¯ Z i + 1 , p Z i = p ¯ Q ) ≤ D ( Q ) . Proof. Our proof is based on the induction method using the base case in Theorem 1. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 16 / 23

4. Applications Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 17 / 23

Finding optimal binary partition N = 2 Based on Theorem 1, � z 1 if r y ≤ a ∗ 1 , Q ∗ ( r y ) = z 2 if r y > a ∗ 1 , for an optimal a ∗ 1 ∈ ( 0 , 1 ) . • The optimal quantizer can be found by an exhaustive searching over a ∗ 1 ∈ ( 0 , 1 ) . • If φ 2 ( y ) φ 1 ( y ) is a strictly increasing/decreasing function, the classical one threshold quantizer is optimal. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 18 / 23

Finding optimal quantizer for N > 2 From Theorem 2, finding the optimal quantizer is equivalent to finding N + 1 scalar thresholds a 0 = 0 < a 1 < · · · < a N − 1 < a N = 1 such that Q ( y ) = z i , if a i − 1 ≤ r y = p x 1 | y < a i . → The problem of finding globally optimal quantizer can be cast as a 1-dimensional scalar quantization problem that can be solved efficiently using the well-known dynamic programming [1], [11], [19]. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 19 / 23

5. Conclusion Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 20 / 23

In this paper , • We show that the optimal quantizer that maximizes the mutual information between input and output under an output constraint separates r y into convex cells. • We provide a sufficient condition for which a single threshold quantizer is optimal. • Some fast algorithms are proposed for determining the optimal quantizers. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 21 / 23

Thank you for listening !!! Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 22 / 23