FUNDAMENTAL PERFORMANCE LIMITS OF ANALOG-TO-DIGITAL COMPRESSION - PowerPoint PPT Presentation

FUNDAMENTAL PERFORMANCE LIMITS OF ANALOG-TO-DIGITAL COMPRESSION Alon Kipnis Stanford University Yonina Eldar Tsachy Weissman Andrea Goldsmith Technion Stanford University Stanford University April 2016 1

OUTLINE Motivation: fundamental performance limits of analog-to-digital compression Combined sampling and source coding Optimal sampling structure Optimal sampling frequency under bitrate constraint Below the Nyquist rate 2

MOTIVATION: Analog to digital (A/D) conversion: 010010011001 001000010000 1000100111… Main goal: measuring and minimizing distortion in A/D D error R bitrate(memory) [bit/sec] 3

MOTIVATION COMBINED SAMPLING AND SOURCE CODING Analog source coding: W ∈ { 0 , 1 } b RT c ˆ X (0 : T ) Encoder Decoder X (0 : T ) ? 01001001 00101001 Encoder f s { 0 , 1 } b RT c X (0 : T ) Quantizer Distortion is due to: Time discretization / sampling (next slide) Finite bit representation (lossy compression) System model: combined sampling and source coding Y [ · ] R f s ˆ X ( · ) Encoder Decoder X ( · ) 4

MOTIVATION WHY SUB-SAMPLING? f s Y [ · ] R ˆ X ( · ) Encoder Decoder X ( · ) Why sub-sampling? Sampling rate is limited by technology High sampling rate bloats memory 24 P=9.13x10 11 Data from 1978 to 1999 22 Data from 2000 to 2005 20 SNRbits (effective number of bits) P=4.1x10 11 Analog Devices: 18 24 bit 2.5MS/s 16 16 bit 100 MS/s 14 12 10 8 6 4 2005 2 1999 0 0 2 4 6 8 10 10 10 10 10 10 10 5 F sample (HZ)

MOTIVATION WHY SUB-SAMPLING? Sampling theorem: f s > f Nyq , 2 f B ◆ sin ( f s t − n ) ✓ n X X ( t ) = X f s f s t − n n ∈ Z D ( R ) t Shannon [1948]: “we are not interested in exact transmission when we have a continuous source, but only in transmission R to within a given tolerance” D(R) is the minimal distortion possible using bitrate R Can we achieve D(R) by sampling below f Nyq ? 6

PROBLEM FORMULATION INFORMATION THEORETIC REPRESENTATION Y [0 : b Tf s c ] f s n 1 , . . . , 2 b T R c o ˆ X (0 : T ) Encoder Decoder X (0 : T ) d ( x ( t ) , ˆ x ( t )) Remote (indirect) source coding [Dobrushin [Wolf & Ziv ’70], [Berger ’71] [Witsenhausen ’80 CEO… I ( Y ; ˆ X ) ≤ R Y [ · ] f s ˆ X ( · ) P Y | ˆ X ( · ) X Z T 1 D ( f s , R ) = inf inf E d ( x ( t ) , ˆ x ( t )) dt 2 T T Y → ˆ X − T I ( Y ; ˆ X ) ≤ R 7

BACKGROUND REMOTE SOURCE CODING ⇣ ⌘ X n , ˆ Y m ( n ) d X n 1 , . . . , 2 nR � 0 X n P Y|X Enc Dec ˆ 0 X n 0 Reduced distortion measure [Witsenhausen ’80] h i ˆ 0 , ˆ 0 , ˆ 0 ) , E d ( y m 0 , ˆ d ( X n 0 | Y m = y m = y m y m X n Y m 0 0 0 ⇣ ⌘ ⇣ ⌘ 0 , ˆ = E ˆ 0 , ˆ X n X n Y m Y m E d n d m 0 0 1 , . . . , 2 nR � Y m ( n ) ˆ Y m Enc Dec 0 0 Corollary: source coding theorem: ⇣ ⌘ 0 , ˆ 0 ( ˆ X n X n Y m D X | Y ( R ) = inf inf 0 ) E d n n Y → ˆ Y I ( Y, ˆ Y ) ≤ R 8

PROBLEM FORMULATION ASSUMPTIONS I ( Y ; ˆ X ) ≤ R Y [ · ] f s ˆ X ( · ) P Y | ˆ X ( · ) X x ( t )) 2 d ( x ( t ) , ˆ x ( t )) = ( x ( t ) − ˆ quadratic distortion [Wolf & Ziv ‘70] Z T 1 ⌘ 2 ⇣ E [ X ( t ) | Y ] − ˆ D ( f s , R ) = mmse X | Y ( f s ) + inf lim X ( t ) dt ⇒ 2 T T →∞ Y → ˆ X − T I ( Y ; ˆ X ) ≤ R X ( · ) is stationary Gaussian with PSD S X ( f ) S X ( f ) S X ( f ) is unimodal f pointwise uniform sampler Y [ n ] = X ( n/f s ) 9

SPECIAL CASE: QUADRATIC GAUSSIAN DISTORTION-RATE f s > f Nyq f s Y [ · ] R ˆ Encoder Decoder X ( · ) X ( · ) D ( f s , R ) = D ( R ) [Pinsker1954] Z ∞ R ( θ ) = 1 log + [ S X ( f ) / θ ] d f 2 −∞ Z ∞ D ( θ ) = min { S X ( f ) , θ } d f −∞ ⇥ S X ( f ) S X ( f ) D ( R ) θ θ 10 f f R

RESULT I RESULT I: SAMPLING-RATE-DISTORTION FUNCTION f s Y [ · ] R ˆ X ( · ) Encoder Decoder X ( · ) Theorem [K., Goldsmith, Weissman, Eldar 2014] Z log + h i fs R ( f s , θ ) = 1 2 e S X | Y ( f ) / θ d f 2 − fs Z fs 2 2 min { e D ( f s , θ ) = mmse X | Y ( f s ) + S X | Y ( f ) , θ } d f − fs 2 Z fs 2 e mmse X | Y ( f s ) = σ 2 S X | Y ( f ) d f X − − fs P 2 k ∈ Z S 2 X ( f − f s k ) e P S X | Y ( f ) = k ∈ Z S X ( f − f s k ) 11

RESULT I WATERFILLING INTERPRETATION Z log + h i fs R ( f s , θ ) = 1 2 e S X | Y ( f ) / θ d f 2 − fs Z fs 2 2 min { e D ( f s , θ ) = mmse X | Y ( f s ) + S X | Y ( f ) , θ } d f − fs 2 D ( f s , R ) = mmse + waterfilling Distortion due X S X ( f − f s k ) fs Z 2 to sampling X σ 2 X = S X ( f − f s k ) d f k ∈ Z e S X | Y ( f ) − fs k ∈ Z 2 Z fs 2 e mmse X | Y ( f s ) = σ 2 S X | Y ( f ) d f X − − fs 2 Distortion due θ to rate constraint f s 12 f

RESULT I EXAMPLE D ( f s , R ) vs f s ( R = 1) 1 S X ( f ) D ( R , f s ) Distortion f W D ( R ) 1 4 f s f Nyq ( 2 W 2 − 2 R f s f s 1 − 2 W + f s < 2 W fs D ( f s , R ) = 2 − R/W f s ≥ 2 W 13

RESULT II EXTENDED MODEL: PRE-SAMPLING FILTER f s Y [ · ] R ˆ X ( · ) H(f) Encoder Decoder X ( · ) Theorem [K. Goldsmith, Weissman, Eldar 2014] Z log + h i fs R ( f s , θ ) = 1 2 e S X | Y ( f ) / θ d f 2 Z − fs n o fs 2 2 e D ( f s , θ , H ) = mmse X | Y ( f s ) + min S X | Y ( f ) , θ d f − fs 2 X ( f − f s k ) | H ( f − f s k ) | 2 k ∈ Z S 2 P ˜ S X | Y ( f ) = k ∈ Z S X ( f − f s k ) | H ( f − f s k ) | 2 P What is D ? ( R, f s ) , inf H D ( R, f s , H ) ? 14

RESULT II RESULT II: OPTIMAL PRE-SAMPLING FILTER Theorem [K. Goldsmith, Eldar, Weissman 2014] Optimal pre-sampling filter maximizes passband energy under an aliasing-free constraint. For unimodal PSD: fs R ( θ ) = 1 Z 2 log + [ S X ( f ) / θ ] d f 2 − fs fs 2 Z 2 D ? ( f s , θ ) = mmse ? X | Y ( f s ) + min { S X ( f ) , θ } d f − fs 2 Suppresses lower energy bands S X ( f ) When PSD is non-unimodal H ? ( f ) filter-bank sampling must be used θ f s 15 15 − f s f s 2 2

RESULT II WHY ANTI-ALIASING IS OPTIMAL? f s Y [ · ] X ( · ) H(f) X 1 ∼ N (0 , σ 2 1 ) e h 1 X 1 , E [ X 1 | Y ] Y = h 1 X 1 + h 2 X 2 + mmse X 2 ∼ N (0 , σ 2 2 ) h 2 e X 2 , E [ X 2 | Y ] ⇣ ⌘ 2 ⇣ ⌘ 2 = h 2 1 σ 4 1 + h 2 2 σ 4 X 1 − f X 2 − f 2 mmse ( X 1 ,X 2 ) | Y = X 1 X 2 + h 2 1 σ 2 1 + h 2 2 σ 2 2 S X ( f ) σ 2 1 , σ 2 � h 1 ,h 2 mmse ( X 1 ,X 2 ) | Y = min inf 2 H ? ( f ) ( σ 2 1 < σ 2 h 1 = 0 , h 2 = 1 , 2 f s σ 2 1 > σ 2 h 1 = 1 , h 2 = 0 , 16 2

RESULT II OPTIMAL PRE-SAMPLING FILTER - EXAMPLE f s Y [ · ] R ˆ X ( · ) H(f) Encoder Decoder X ( · ) D ( R, f s ) D ? ( R, f s ) and D ( R, f s ) vs f s 1 0.9 0.8 θ Distortion f s 0.7 f D ( R, f s ) 0.6 D ? ( R, f s ) 0.5 D ? ( R H ? ( f ) 0.4 , f s ) 0.3 0.2 θ f s 0.1 f s 0 0.5 1 1.5 2 − f s f s 2 2

COROLLARY: OPTIMAL SAMPLING RATE S X ( f ) S X ( f ) S X ( f ) θ θ θ f s f s f s D ? ( R, f s ) vs f s D ? ( f s , R ) D ( R ) f s 0 0.5 1 1.5 2 f Ny f DR ( R ) D ? ( f s , R ) = D ( R ) for f s ≥ f DR ( R ) (!) 18

RESULT III: RESULT III: OPTIMAL SAMPLING RATE IN ANALOG-TO-DIGITAL COMPRESSION A new sampling theorem for Gaussian stationary processes: Theorem [K. Goldsmith, Eldar 2015] D ? ( f s , R ) = D ( R ) f s ≥ f DR ( R ) Extends Shannon-Nyquist-Kotelinkov- Whittaker sampling theorem: Incorporates lossy compression S X ( f ) Valid when X() is not bandlimited H ? ( f ) Holds under non-uniform sampling Interpretation: lossy compression θ reduces degrees of freedom f DR ( R )

RESULT III OPTIMAL SAMPLING RATE - EXAMPLE S X ( f ) f s D ? ( f s , R ) as a function of f Nyq f DR ( R ) σ 2 X D ? ( f s , R ) [dB] − 2 − 1.5 − 1 − 0.5 0 0.5 1 1.5 2 R = 0 . 5 R = 1 R = 2 f s R = 20 f Nyq 1 20

RESULT III OPTIMAL SAMPLING RATE - EXAMPLE 21

SAMPLING NON-BANDLIMITED SIGNALS S X ( f ) f Nyq = ∞ H ? ( f ) f DR ( R ) < ∞ θ MSE [dB] f s f − f s f s 2 2 f DR ( R ) f DR ( R ) → ∞ f s Sampling a Brownian motion ? ISIT 2016 …. 22 22

SUMMARY Constraining sampling frequency in the analog source coding problem leads to D(fs,R) = mmse + `water-filling’ Optimal pre-sampling filter eliminates aliasing New critical sampling frequency f performance D(R) under bitrate constraints — typically below the Nyquist rate 23

THE END! D ( R, f s ) θ f s f 24

OPEN QUESTIONS AND FUTURE WORK Optimal pre-sampling filter with different loss criterion Unknown source statistic Effect of lossy compression on DoF in other signal model (e.g. sparse signals) Optimal sampling rate - bit allocation strategy in existing A/D schemes Linear pre-processing reduces sampling distortion. Can non-linear (time-preserving) 25

REFERENCES A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, “Rate distortion of Gaussian stationary processes” 2015, available online A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Sub-Nyquist sampling achieves optimal rate-distortion” A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Distrotion-rate function under sub-Nyquist nonuniform samplig” A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Rate-Distortion Function of Cyclostationary Gaussian Processes” A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Optimal tradeoff between sampling rate and quantizer resolution in A/D conversion” 26