On Third-Order Asymptotics for DMCs Vincent Y. F. Tan Institute for - PowerPoint PPT Presentation

On Third-Order Asymptotics for DMCs Vincent Y. F. Tan Institute for Infocomm Research (I 2 R) National University of Singapore (NUS) January 20, 2013 Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 1 / 29

Acknowledgements This is joint work with Marco Tomamichel Centre for Quantum Technologies National University of Singapore Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 2 / 29

Transmission of Information INFORMATION RECEIVER DESTINATION TRANSMITTER SOURCE RECEIVED SIGNAL SIGNAL MESSAGE MESSAGE NOISE SOURCE Shannon’s Figure 1 Information theory ≡ Finding fundamental limits for reliable information transmission Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 3 / 29

Transmission of Information INFORMATION RECEIVER DESTINATION TRANSMITTER SOURCE RECEIVED SIGNAL SIGNAL MESSAGE MESSAGE NOISE SOURCE Shannon’s Figure 1 Information theory ≡ Finding fundamental limits for reliable information transmission Channel coding: Concerned with the maximum rate of communication in bits/channel use Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 3 / 29

Channel Coding (One-Shot) � M X Y M ✲ ✲ ✲ ✲ e W d A code is an triple C = {M , e , d } where M is the message set Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

Channel Coding (One-Shot) � M X Y M ✲ ✲ ✲ ✲ e W d A code is an triple C = {M , e , d } where M is the message set The average error probability p err ( C ) is p err ( C ) := Pr [ � M � = M ] where M is uniform on M Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

Channel Coding (One-Shot) � M X Y M ✲ ✲ ✲ ✲ e W d A code is an triple C = {M , e , d } where M is the message set The average error probability p err ( C ) is p err ( C ) := Pr [ � M � = M ] where M is uniform on M ε -Error Capacity is � � � � ∃ C s.t. m = |M| , p err ( C ) ≤ ε M ∗ ( W , ε ) := sup m ∈ N Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

Channel Coding ( n -Shot) � X n Y n M M ✲ ✲ ✲ ✲ W n e d Consider n independent uses of a channel Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

Channel Coding ( n -Shot) � X n Y n M M ✲ ✲ ✲ ✲ W n e d Consider n independent uses of a channel Assume W is a discrete memoryless channel Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

Channel Coding ( n -Shot) � X n Y n M M ✲ ✲ ✲ ✲ W n e d Consider n independent uses of a channel Assume W is a discrete memoryless channel For vectors x = ( x 1 , . . . , x n ) ∈ X n and y := ( y 1 , . . . , y n ) ∈ Y n , n � W n ( y | x ) = W ( y i | x i ) i = 1 Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

Channel Coding ( n -Shot) � X n Y n M M ✲ ✲ ✲ ✲ W n e d Consider n independent uses of a channel Assume W is a discrete memoryless channel For vectors x = ( x 1 , . . . , x n ) ∈ X n and y := ( y 1 , . . . , y n ) ∈ Y n , n � W n ( y | x ) = W ( y i | x i ) i = 1 Blocklength n , ε -Error Capacity is M ∗ ( W n , ε ) Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

Main Contribution Upper bound log M ∗ ( W n , ε ) for n large (converse) Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

Main Contribution Upper bound log M ∗ ( W n , ε ) for n large (converse) Concerned with the third-order term of the asymptotic expansion Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

Main Contribution Upper bound log M ∗ ( W n , ε ) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

Main Contribution Upper bound log M ∗ ( W n , ε ) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms Theorem (Tomamichel-Tan (2013)) For all DMCs with positive ε -dispersion V ε , √ nV ε Q − 1 ( ε ) + 1 log M ∗ ( W n , ε ) ≤ nC − 2 log n + O ( 1 ) � + ∞ � 2 x 2 � 1 − 1 where Q ( a ) := 2 π exp d x √ a Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

Main Contribution Upper bound log M ∗ ( W n , ε ) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms Theorem (Tomamichel-Tan (2013)) For all DMCs with positive ε -dispersion V ε , √ nV ε Q − 1 ( ε ) + 1 log M ∗ ( W n , ε ) ≤ nC − 2 log n + O ( 1 ) � + ∞ � 2 x 2 � 1 − 1 where Q ( a ) := 2 π exp d x √ a The 1 2 log n term is our main contribution Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

Main Contribution: Remarks Our bound √ nV ε Q − 1 ( ε ) + 1 log M ∗ ( W n , ε ) ≤ nC − 2 log n + O ( 1 ) Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

Main Contribution: Remarks Our bound √ nV ε Q − 1 ( ε ) + 1 log M ∗ ( W n , ε ) ≤ nC − 2 log n + O ( 1 ) Best upper bound till date: � � √ |X| − 1 log M ∗ ( W n , ε ) ≤ nC − nV ε Q − 1 ( ε ) + log n + O ( 1 ) 2 V. Strassen (1964) Polyanskiy-Poor-Verdú or PPV (2010) Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

Main Contribution: Remarks Our bound √ nV ε Q − 1 ( ε ) + 1 log M ∗ ( W n , ε ) ≤ nC − 2 log n + O ( 1 ) Best upper bound till date: � � √ |X| − 1 log M ∗ ( W n , ε ) ≤ nC − nV ε Q − 1 ( ε ) + log n + O ( 1 ) 2 V. Strassen (1964) Polyanskiy-Poor-Verdú or PPV (2010) Requires new converse techniques Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

Outline 1 Background 2 Related work 3 Main result 4 New converse 5 Proof sketch 6 Summary and open problems Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 8 / 29

Background: Shannon’s Channel Coding Theorem Shannon’s noisy channel coding theorem and Wolfowitz’s strong converse state that Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 9 / 29

Background: Shannon’s Channel Coding Theorem Shannon’s noisy channel coding theorem and Wolfowitz’s strong converse state that Theorem (Shannon (1949), Wolfowitz (1959)) 1 n log M ∗ ( W n , ε ) = C , lim ∀ ε ∈ ( 0 , 1 ) n →∞ where C is the channel capacity defined as C = C ( W ) = max I ( P , W ) P Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 9 / 29

Background: Shannon’s Channel Coding Theorem 1 n log M ∗ ( W n , ε ) = C lim bits/channel use n →∞ Noisy channel coding theorem is independent of ε ∈ ( 0 , 1 ) Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

Background: Shannon’s Channel Coding Theorem 1 n log M ∗ ( W n , ε ) = C lim bits/channel use n →∞ Noisy channel coding theorem is independent of ε ∈ ( 0 , 1 ) n →∞ p err ( C ) lim ✻ 1 ✲ R 0 C Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

Background: Shannon’s Channel Coding Theorem 1 n log M ∗ ( W n , ε ) = C lim bits/channel use n →∞ Noisy channel coding theorem is independent of ε ∈ ( 0 , 1 ) n →∞ p err ( C ) lim ✻ 1 ✲ R 0 C Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

Background: Shannon’s Channel Coding Theorem 1 n log M ∗ ( W n , ε ) = C lim bits/channel use n →∞ Noisy channel coding theorem is independent of ε ∈ ( 0 , 1 ) n →∞ p err ( C ) lim ✻ 1 ✲ R 0 C Phase transition at capacity Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

Background: ε -Dispersion What happens at capacity? Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

Background: ε -Dispersion What happens at capacity? More precisely, what happens when log |M| ≈ nC + a √ n for some a ∈ R ? Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

Background: ε -Dispersion What happens at capacity? More precisely, what happens when log |M| ≈ nC + a √ n for some a ∈ R ? Assume capacity-achieving input distribution (CAID) P ∗ is unique Vincent Tan (I 2 R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29