Optimum Source Resolvability Rate with Respect to f -Divergences - PowerPoint PPT Presentation

Optimum Source Resolvability Rate with Respect to f -Divergences Using the Smooth Rényi Entropy Ryo Nomura (Waseda University, Japan) Hideki Yagi (The Universtiy of Electro-Communications, Japan) ISIT2020, June 21-26, 2020

Outline • Preliminaries 1. Source resolvability 2. f -divergence 3. Smooth Rényi entropy • Main results (Optimum D -achievable resolvability rate ) • Specifjcations • Conclusion 1

Preliminaries

1. Preliminaries We focus on the source resolvability problem. Given an arbitrary target source, we approximate it by using a discrete random variable which is uniformly distributed. Mapping φ : U M := { 1 , 2 , . . . , M } → X Uniform Random Number Given Target Source i P ( i ) φ x P ( x ) 1 1 /M → 1 1 / 3 2 1 /M X : 2 1 / 8 U M : 3 1 /M 3 1 / 9 · · · · · · · · · · · · M 1 /M 2

1. Preliminaries We focus on the source resolvability problem. Given an arbitrary target source, we approximate it by using a discrete random variable which is uniformly distributed. Mapping φ : U M := { 1 , 2 , . . . , M } → X Uniform Random Number Given Target Source i P ( i ) φ x P ( x ) 1 1 /M → 1 1 / 3 2 1 /M X : 2 1 / 8 U M : 3 1 /M 3 1 / 9 · · · · · · · · · · · · M 1 /M 2

1. Preliminaries • Distance d ( φ ( U M ) , X ) : requested to be small • Size M (rate: log M ): requested to be small 3

1. Preliminaries • Distance d ( φ ( U M ) , X ) : requested to be small • Size M (rate: log M ): requested to be small 3

1. Preliminaries • Distance d ( φ ( U M ) , X ) : requested to be small ( ≤ D ) • Size M (rate: log M ): requested to be as small as possible 4

1. Preliminaries Table 1: Previous results Approximation measure (or distance) Variational distance KL-divergence f -divergence Han & Verdú, ’93 Information Steinberg & Verdú, ’96 Nomura, Steinberg & Verdú, ’96 Spectrum Nomura, ’18 ISIT’19 Yagi & Han, ’17 Rényi Uyematsu, ’10 Entropy 5

1. Preliminaries Table 2: Previous results Approximation measure (or distance) Variational distance KL-divergence f -divergence Han & Verdú, ’93 Information Steinberg & Verdú, ’96 Nomura, Steinberg & Verdú, ’96 Spectrum Nomura, ’18 ISIT’19 Yagi & Han, ’17 Rényi Uyematsu, ’10 This study Entropy 6

1. Preliminaries Notation • X = { X n } ∞ n =1 : general source with values in countable sets X n . • P X n : probability distribution of X n • U M : random variable uniformly distributed on U M := { 1 , 2 , · · · , M } , P U M ( i ) = 1 M , 1 ≤ i ≤ M Assumption: � � � 1 � � 1 � H ( X ) := sup R � lim n →∞ Pr n log P X n ( X n ) ≥ R = 1 < + ∞ � 7

1. Preliminaries Notation • X = { X n } ∞ n =1 : general source with values in countable sets X n . • P X n : probability distribution of X n • U M : random variable uniformly distributed on U M := { 1 , 2 , · · · , M } , P U M ( i ) = 1 M , 1 ≤ i ≤ M Assumption: � � � 1 � � 1 � H ( X ) := sup R � lim n →∞ Pr n log P X n ( X n ) ≥ R = 1 < + ∞ � 7

1. Preliminaries We defjne a class of f -divergences between P Z and P Z . Let f ( t ) be a convex function defjned for t > 0 and f (1) = 0 . Defjnition ([Csiszár and Shields, ’04]) Let P Z and P Z denote probability distributions over a fjnite set Z . The f -divergence between P Z and P Z is defjned by � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f , P Z ( z ) z ∈Z � 0 where we set 0 f = 0 , f (0) = lim t → 0 f ( t ) , 0 f ( a f ( u ) u . � 0 ) = a lim u →∞ 0 We next give some examples of f -divergences. 8

1. Preliminaries We defjne a class of f -divergences between P Z and P Z . Let f ( t ) be a convex function defjned for t > 0 and f (1) = 0 . Defjnition ([Csiszár and Shields, ’04]) Let P Z and P Z denote probability distributions over a fjnite set Z . The f -divergence between P Z and P Z is defjned by � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f , P Z ( z ) z ∈Z � 0 where we set 0 f = 0 , f (0) = lim t → 0 f ( t ) , 0 f ( a f ( u ) u . � 0 ) = a lim u →∞ 0 We next give some examples of f -divergences. 8

1. Preliminaries We defjne a class of f -divergences between P Z and P Z . Let f ( t ) be a convex function defjned for t > 0 and f (1) = 0 . Defjnition ([Csiszár and Shields, ’04]) Let P Z and P Z denote probability distributions over a fjnite set Z . The f -divergence between P Z and P Z is defjned by � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f , P Z ( z ) z ∈Z � 0 where we set 0 f = 0 , f (0) = lim t → 0 f ( t ) , 0 f ( a f ( u ) u . � 0 ) = a lim u →∞ 0 We next give some examples of f -divergences. 8

1. Preliminaries We defjne a class of f -divergences between P Z and P Z . Let f ( t ) be a convex function defjned for t > 0 and f (1) = 0 . Defjnition ([Csiszár and Shields, ’04]) Let P Z and P Z denote probability distributions over a fjnite set Z . The f -divergence between P Z and P Z is defjned by � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f , P Z ( z ) z ∈Z � 0 where we set 0 f = 0 , f (0) = lim t → 0 f ( t ) , 0 f ( a f ( u ) u . � 0 ) = a lim u →∞ 0 We next give some examples of f -divergences. 8

1. Preliminaries Defjnition � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f P Z ( z ) z ∈Z Examples [Csiszár and Shields, ’04][Sason and Verdú, ’16] • f ( t ) = t log t : (KL divergence) P Z ( z ) log P Z ( z ) � D f ( Z || Z ) = P Z ( z ) =: D ( Z || Z ) . z ∈Z • f ( t ) = − log t : (Reverse KL divergence) P Z ( z ) log P Z ( z ) � D f ( Z || Z ) = P Z ( z ) = D ( Z || Z ) . 9 z ∈Z

1. Preliminaries Defjnition � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f . P Z ( z ) z ∈Z Examples [Csiszár and Shields, ’04][Sason and Verdú, ’16] √ • f ( t ) = 1 − t : (Hellinger distance) � � D f ( Z || Z ) = 1 − P Z ( z ) P Z ( z ) . z ∈Z • f ( t ) = ( t − 1) + = (1 − t ) + := max { 1 − t, 0 } : (Variational distance) D f ( Z || Z ) = 1 � | ( P Z ( z ) − P Z ( z )) | . 2 z ∈Z 10

1. Preliminaries Defjnition � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f . P Z ( z ) z ∈Z Examples [Csiszár and Shields, ’04][Sason and Verdú, ’16] • f ( t ) = ( t − γ ) + : ( E γ -divergence) For any given γ ≥ 1 , � D f ( Z || Z ) = ( P Z ( z ) − γP Z ( z )) =: E γ ( Z || Z ) . z ∈Z : P Z ( z ) >γP Z ( z ) • It is not diffjcult to check that f ( t ) = ( γ − t ) + + 1 − γ leads E γ -divergence. • γ = 1 : variational distance 11

1. Preliminaries Defjnition � P Z ( z ) � � D f ( Z || Z ) := P Z ( z ) f . P Z ( z ) z ∈Z Examples [Csiszár and Shields, ’04][Sason and Verdú, ’16] • f ( t ) = ( t − γ ) + : ( E γ -divergence) For any given γ ≥ 1 , � D f ( Z || Z ) = ( P Z ( z ) − γP Z ( z )) =: E γ ( Z || Z ) . z ∈Z : P Z ( z ) >γP Z ( z ) • It is not diffjcult to check that f ( t ) = ( γ − t ) + + 1 − γ leads E γ -divergence. • γ = 1 : variational distance 11

1. Preliminaries In this study, we assume the following conditions on the function f . C1) The function f ( t ) is a strictly decreasing function of t for t ∈ (0 , 1] and a decreasing function for t ≥ 1 . C2) For any pair of positive numbers ( a, b ) , it holds that � e − nb � f lim = 0 . e na n →∞ C3) For any number a ∈ [0 , 1] , it holds that � a � 0 f = 0 . 0 Note We only consider the f -divergence with the function f satisfying C1)–C3). 12

1. Preliminaries In this study, we assume the following conditions on the function f . C1) The function f ( t ) is a strictly decreasing function of t for t ∈ (0 , 1] and a decreasing function for t ≥ 1 . C2) For any pair of positive numbers ( a, b ) , it holds that � e − nb � f lim = 0 . e na n →∞ C3) For any number a ∈ [0 , 1] , it holds that � a � 0 f = 0 . 0 Note We only consider the f -divergence with the function f satisfying C1)–C3). 12

1. Preliminaries In this study, we assume the following conditions on the function f . C1) The function f ( t ) is a strictly decreasing function of t for t ∈ (0 , 1] and a decreasing function for t ≥ 1 . C2) For any pair of positive numbers ( a, b ) , it holds that � e − nb � f lim = 0 . e na n →∞ C3) For any number a ∈ [0 , 1] , it holds that � a � 0 f = 0 . 0 Note We only consider the f -divergence with the function f satisfying C1)–C3). 12

1. Preliminaries In this study, we assume the following conditions on the function f . C1) The function f ( t ) is a strictly decreasing function of t for t ∈ (0 , 1] and a decreasing function for t ≥ 1 . C2) For any pair of positive numbers ( a, b ) , it holds that � e − nb � f lim = 0 . e na n →∞ C3) For any number a ∈ [0 , 1] , it holds that � a � 0 f = 0 . 0 Note We only consider the f -divergence with the function f satisfying C1)–C3). 12

1. Preliminaries 13

2. Main result Note √ t , and f ( t ) = (1 − t ) + satisfy three conditions, f ( t ) = − log t , f ( t ) = 1 − while f ( t ) = t log t does not satisfy C1). 14

1. Preliminaries Note f ( t ) = ( γ − t ) + + 1 − γ satisfy three conditions, while f ( t ) = ( t − γ ) + does not satisfy C1). 15