On the Compressibility of Affinely Singular Random Vectors Mohammad - PowerPoint PPT Presentation

On the Compressibility of Affinely Singular Random Vectors Mohammad Amin Charusaie † , Stefano Rini ∗ , Arash Amini † † Sharif University of Technology, ∗ National Chiao Tung University Full version: https://arxiv.org/abs/2001.03884 amin.ch90@gmail.com June 8, 2020 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 1 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy Shannon Entropy : minimum bits needed to transmit a discrete source M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy Shannon Entropy : minimum bits needed to transmit a discrete source The R´ enyi information dimension (RID): Shannon compressibility measure defined for general measures M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy Shannon Entropy : minimum bits needed to transmit a discrete source The R´ enyi information dimension (RID): Shannon compressibility measure defined for general measures minimum bits to transmit X with high fidelity RID ( X ) = minimum bits to transmit any 1D source with the same fidelity M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy Shannon Entropy : minimum bits needed to transmit a discrete source The R´ enyi information dimension (RID): Shannon compressibility measure defined for general measures minimum bits to transmit X with high fidelity RID ( X ) = minimum bits to transmit any 1D source with the same fidelity RID for absolutely continuous and discrete RVs ⇒ known M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

A quick overview of the problem Compressibility measures of random vectors (RVs) based on Shannon’s entropy Shannon Entropy : minimum bits needed to transmit a discrete source The R´ enyi information dimension (RID): Shannon compressibility measure defined for general measures minimum bits to transmit X with high fidelity RID ( X ) = minimum bits to transmit any 1D source with the same fidelity RID for absolutely continuous and discrete RVs ⇒ known But, RID for measures with singularity, RID-based compressibility measures for stochastic processes (SPs) with singularity, and Relationship between the RID and other compressibility measures are yet to be determined. M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 2 / 22

What this work is about? A class of SPs M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes Linear transformation of independent DC RVs M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes Linear transformation of independent DC RVs RID of Affinely singular random vectors M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes Linear transformation of independent DC RVs RID of Affinely singular random vectors A linear transformation of orthogonally singular random vectors (which include independent DC RVs) M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes Linear transformation of independent DC RVs RID of Affinely singular random vectors A linear transformation of orthogonally singular random vectors (which include independent DC RVs) RID-based compressibility measures for these MA processes M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

What this work is about? A class of SPs Passing a discrete-continuous (DC) white noise from an FIR filter A subclass of moving-average (MA) processes Linear transformation of independent DC RVs RID of Affinely singular random vectors A linear transformation of orthogonally singular random vectors (which include independent DC RVs) RID-based compressibility measures for these MA processes RID-based compressibility measures measures = ǫ -achievable rate M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 3 / 22

Contents Affinely singular measures and its applications 1 RID of random variables and processes 2 RID of affinely singular RVs 3 Information dimension and ǫ -achievale rates of discrete-continuous 4 moving-average processes Conclusion 5 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 4 / 22

Types of Measures Definition 1 The measure µ ( · ) is Singular: S ∈ R n , L´ ebesgue Measure of S = 0 , µ ( S ) > 0 . 1 W. Rudin, ”Real and complex analysis,” 2006, pp. 121 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 5 / 22

Types of Measures Definition 1 The measure µ ( · ) is Singular: S ∈ R n , L´ ebesgue Measure of S = 0 , µ ( S ) > 0 . Absolutely continuous: no such subset exists 1 W. Rudin, ”Real and complex analysis,” 2006, pp. 121 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 5 / 22

Types of Measures Definition 1 The measure µ ( · ) is Singular: S ∈ R n , L´ ebesgue Measure of S = 0 , µ ( S ) > 0 . Absolutely continuous: no such subset exists Discrete: µ ( · ) supported on a countable set 1 W. Rudin, ”Real and complex analysis,” 2006, pp. 121 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 5 / 22

Types of Measures Definition 1 The measure µ ( · ) is Singular: S ∈ R n , L´ ebesgue Measure of S = 0 , µ ( S ) > 0 . Absolutely continuous: no such subset exists Discrete: µ ( · ) supported on a countable set Discrete-continuous: µ ( · ) convex combination of discrete and absolutely continuous measures in 1D 1 W. Rudin, ”Real and complex analysis,” 2006, pp. 121 M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 5 / 22

Affinely Singular Measures Definition Z m = � Z ( i ) ⇒ Affinely Singular RV : i ∈ N 1 { V = i } � Z ( i ) = U i [ X c,i ; 0 m − h i ] + b i , � V ⇒ random choice of Z m , U i : m × m unitary matrix, 0 k : k -dimensional column vector of all zeros, b i : fixed vector in R m , and X c,i : h i -dimensional absolutely continuous RV. M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 6 / 22

Orthogonally Singular Measures Definition X n : orthogonally singular RV X i = ν i X c i + (1 − ν i ) X d i X n c : absolutely continuous RV, X n d : discrete RV, Bernoulli ν i with P ( ν i = 1) = α i ⇒ random choice of X n , and X n c independent of X n d and ν . 2 See Lemma 6 of Full version of the paper M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 7 / 22

Orthogonally Singular Measures Definition X n : orthogonally singular RV X i = ν i X c i + (1 − ν i ) X d i X n c : absolutely continuous RV, X n d : discrete RV, Bernoulli ν i with P ( ν i = 1) = α i ⇒ random choice of X n , and X n c independent of X n d and ν . X c i s, X d i s, and ν i s are mutually independent ⇒ X n is a random vector with independent DC RVs. 2 See Lemma 6 of Full version of the paper M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 7 / 22

Orthogonally Singular Measures Definition X n : orthogonally singular RV X i = ν i X c i + (1 − ν i ) X d i X n c : absolutely continuous RV, X n d : discrete RV, Bernoulli ν i with P ( ν i = 1) = α i ⇒ random choice of X n , and X n c independent of X n d and ν . X c i s, X d i s, and ν i s are mutually independent ⇒ X n is a random vector with independent DC RVs. Linear transformation of X n has singularities on affine subsets (Affinely Singular RV) 2 2 See Lemma 6 of Full version of the paper M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 7 / 22

An Example: Discrete-Continuous Moving-Average (DC-MA) SPs Consider an FIR filter: H ( s ) = � l 2 i = − l 1 a i s i W m + l 1 + l 2 Y m ( a i s and l i s are fixed constants) M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 8 / 22

An Example: Discrete-Continuous Moving-Average (DC-MA) SPs Consider an FIR filter: H ( s ) = � l 2 i = − l 1 a i s i W m + l 1 + l 2 Y m ( a i s and l i s are fixed constants) W m + l 1 + l 2 RV with i.i.d. DC elements ⇒ Y m samples of a DC-MA SPs M.A. Charusaie, S.Rini, A. Amini ISIT 2020 June 8, 2020 8 / 22