Heat Equation Fan Cheng John Hopcroft Center Computer Science and - PowerPoint PPT Presentation

[Pop-up Salon, Maths, SJTU, 2019/01/10] On the Complete Monotonicity of Heat Equation Fan Cheng John Hopcroft Center Computer Science and Engineering Shanghai Jiao Tong University chengfan@sjtu.edu.cn

Overview 𝜖 2 2𝜖𝑦 2 𝑔 𝑦, 𝑢 = 𝜖 𝜖𝑢 𝑔(𝑦, 𝑢) (Heat equation) ℎ 𝑌 + 𝑢𝑎 𝑍 = 𝑌 + 𝑢𝑎 ℎ 𝑌 = −∫ 𝑦log𝑦 d𝑦 𝑎 ∼ 𝒪(0,1) (Gaussian Channel) CMI = 2 (H. P. McKean, 1966) CMI ≥ 𝟓 . Conjecture: CMI = +∞ (Cheng, 2015)

Outline □ “Super - H” Theorem □ Boltzmann equation and heat equation □ Shannon Entropy Power Inequality □ Complete Monotonicity Conjecture

Fire and Civilization Steam engine Drill Myth: west and east James Watts The Wealth of Nations Independence of US 1776

Study of Heat 𝜖 2 𝜖𝑢 𝑔 𝑦, 𝑢 = 1 𝜖 𝜖𝑦 2 𝑔(𝑦, 𝑢) 2 □ The history begins with the work of Joseph Heat transfer Fourier around 1807 □ In a remarkable memoir, Fourier invented both Heat equation and the method of Fourier analysis for its solution

Information Age Gaussian Channel: 𝑎 𝑢 ∼ 𝒪(0, 𝑢) X and Z are mutually independent. The p.d.f of X is g(x) Y is the convolution of X and : Y = X+ 𝑎 𝑢 𝑎 𝑢 The p.d.f. of Y (𝑧−𝑦) 2 1 𝑔(𝑧; 𝑢) = ∫ 𝑕(𝑦) 𝑓 2𝑢 2𝜌𝑢 A mathematical theory of 𝜖 2 𝜖𝑢 𝑔(𝑧; 𝑢) = 1 𝜖 𝜖𝑧 2 𝑔(𝑧; 𝑢) communication, Bell System Technical 2 Journal. 27 (3): 379 – 423. Fundamentally, Gaussian channel and heat equation are identical in mathematics (Gaussian mixture model)

Entropy Formula Second law of thermodynamics: one way only Entropy

Ludwig Boltzmann Boltzmann formula: 𝑇 = −𝑙 𝐶 ln𝑋 Gibbs formula: 𝑇 = −𝑙 𝑐 ∑ 𝑞 𝑗 ln𝑞 𝑗 𝑗 Boltzmann equation: 𝑒𝑔 𝑒𝑢 = (𝜖𝑔 𝜖𝑢) force + (𝜖𝑔 𝜖𝑢) diff + (𝜖𝑔 Ludwig Eduard Boltzmann 𝜖𝑢) coll 1844-1906 Vienna, Austrian Empire H-theorem: 𝐼(𝑔(𝑢)) is non−decreasing

“Super H - theorem” for Boltzmann Equation □ Notation • A function is completely monotone (CM) iff all the signs of its 1 𝑢 , 𝑓 −𝑢 ) derivatives are: +, -, +, - ,…… (e.g., □ McKean’s Conjecture on Boltzmann equation (1966): □ 𝐼(𝑔(𝑢)) is CM in 𝑢, when 𝑔 𝑢 satisfies Boltzmann equation □ False, disproved by E. Lieb in 1970s □ the particular Bobylev-Krook- Wu explicit solutions, this “theorem” holds true for n ≤ 101 and breaks downs afterwards H. P. McKean, NYU. National Academy of Sciences

“Super H - theorem” for Heat Equation □ Heat equation: Is 𝐼(𝑔(𝑢)) CM in 𝑢 , if 𝑔(𝑢) satisfies heat equation □ Equivalently, is 𝐼(𝑌 + 𝑢𝑎) CM in t? □ The signs of the first two order derivatives were obtained Failed to obtain the 3 rd and 4 th . (It is easy to compute the □ derivatives, it is hard to obtain their signs) “This suggests that……, etc., but I could not prove it” C. Villani, 2010 Fields Medalist

Claude E. Shannon and EPI Central limit theorem Capacity region of Gaussian broadcast channel Capacity region of Gaussian Multiple-Input Multiple-Output broadcast channel Uncertainty principle All of them can be proved by Entropy power inequality (EPI) □ Entropy power inequality (Shannon 1948): For any two independent continuous random variables X and Y, 𝑓 2ℎ(𝑌+𝑍) ≥ 𝑓 2ℎ(𝑌) + 𝑓 2ℎ(𝑍) Equalities holds iff X and Y are Gaussian □ Motivation: Gaussian noise is the worst noise. □ Impact: A new characterization of Gaussian distribution in information theory □ Comments: most profound! (Kolmogorov)

Entropy Power Inequality □ Shannon himself didn’t give a proof but an explanation, which turned out to be wrong □ The first proof is given by A. J. Stam (1959), N. M. Blachman (1966) □ Research on EPI Generalization, new proof, new connection. E.g., Gaussian interference channel is open, some stronger “EPI’’ should exist. □ Stanford Information Theory School: Thomas Cover and his students: A. El Gamel, M. H. Costa, A. Dembo, A. Barron (1980- 1990) □ Princeton Information Theory School: Sergio Verdu, etc. (2000s) Battle field of Shannon theory

Ramification of EPI Gaussian perturbation: ℎ(𝑌 + 𝑢𝑎) Shannon EPI 𝜖 Fisher Information: I 𝑌 + 𝑢𝑎 = 𝜖𝑢 ℎ(𝑌 + 𝑢𝑎)/2 Fisher Information is decreasing in 𝑢 Fisher information inequality (FII): 𝑓 2ℎ(𝑌+ 𝑢𝑎) is concave in 𝑢 1 1 1 𝐽(𝑌+𝑍) ≥ 𝐽(𝑌) + 𝐽(𝑍) Status Quo: FII can imply EPI and all its Tight Young’s inequality generalizations. However, life is always 𝑌 + 𝑍 𝑠 ≥ 𝑑 𝑌 𝑞 𝑍 𝑟 hard. FII is far from enough

On 𝑌 + 𝑢𝑎 𝑌 is arbitrary and ℎ(𝑌) may not exist When t → 0, 𝑌 + 𝑢𝑎 → 𝑌 . When 𝑢 → ∞ , 𝑌 + 𝑢𝑎 → Gaussian When t > 0, 𝑌 + 𝑢𝑎 and ℎ(𝑌 + 𝑢𝑎) are infinitely differential 𝑢𝑎 is called mixed gaussian distribution (Gaussian Mixed Model 𝑌 + (GMM) in machine learning) 𝑢𝑎 is Gaussian channel/source in information theory 𝑌 + Gaussian noise is the worst additive noise Gaussian distribution maximizes ℎ(𝑌) Entropy power inequality, central limit theorem, etc.

Where we take off  Shannon Entropy power inequality Information theorists got lost  Fisher information inequality in the past 70 years  ℎ(𝑌 + 𝑢𝑎) Mathematicians ignored it  ℎ 𝑔 𝑢 is CM  When 𝑔(𝑢) satisfied Boltzmann equation, disproved  When 𝑔(𝑢) satisfied heat equation, unknown  We even don’t know what CM is! 𝒇 −𝒖 𝒂) , Motivation: to study some inequalities; e.g., the convexity of 𝒊(𝒀 + 𝑱 𝒀+ 𝒖𝒂 the concavity of 𝒖 “Any progress?” It is widely believed that there should be no “None…” new EPI except Shannon EPI and FII.

Discovery 𝜖 𝑢𝑎 ≥ 0 (de Bruijn, 1958) 𝐽 𝑌 + 𝑢𝑎 = 2𝜖𝑢 ℎ 𝑌 + 𝐽 (1) = 𝜖 𝑢𝑎 ≤ 0 (McKean1966, Costa 1985) 𝜖𝑢 𝐽 𝑌 + Observation: 𝑱(𝒀 + 𝒖𝒂) is convex in 𝒖 1 1  ℎ 𝑌 + 𝑢 . 𝐽 is CM: +, -, +, - … 𝑢𝑎 = 2 ln 2𝜌𝑓𝑢, 𝐽 𝑌 + 𝑢𝑎 =  If the observation is true, the first three derivatives are: +, -, +  Q: Is the 4 th order derivative -? Because 𝑎 is Gaussian!  The signs of derivatives of ℎ(𝑌 + 𝑢𝑎) are independent of 𝑌 . Invariant!  Exactly the same problem in McKean 1966 To convince people, we must prove its convexity

Challenge Let 𝑌 ∼ 𝑕(𝑦)  ℎ 𝑍 𝑢 = −∫ 𝑔(𝑧, 𝑢) ln 𝑔(𝑧, 𝑢) 𝑒𝑧 , no closed form except some special 𝑕(𝑦) . 𝑔(𝑧, 𝑢) satisfies heat equation. 2 𝑔  𝐽 𝑍 1 𝑢 = ∫ 𝑔 𝑒𝑧 2 2  𝐽 1 𝑍 𝑔 𝑔 2 1 𝑢 = −∫ 𝑔 − 𝑒𝑧 𝑔 2  So what is 𝐽 (2) ? (Heat equation, integration by parts)

Challenge (cont’d) It is trivial to calculate derivatives. It is hard to prove their signs

Breakthrough Integration by parts: ∫ 𝑣𝑒𝑤 = 𝑣𝑤 − ∫ 𝑤𝑒𝑣 First breakthrough since McKean 1966

GCMC Gaussian complete monotonicity conjecture: 𝑱(𝒀 + 𝒖𝒂) is CM in 𝒖 Conjecture: 𝐦𝐩𝐡𝑱(𝒀 + 𝒖𝒂) is convex in 𝒖 Pointed out by C. Villani and G. Toscani the connection with McKean’s paper A general form: number partition. Hard to determine the coefficients. Hard to find 𝛾 𝑙,𝑘 !

Complete monotone function How to construct 𝑕(𝑦) ? A new expression for entropy involved special functions in mathematical physics Herbert R. Stahl, 2013

Complete monotone function  A function f(t) is CM, then logf(t) is convex in t  𝐽 𝑍 𝑢 is CM in t, then log 𝐽(𝑍 𝑢 ) is convex in t  A function f(t) is CM, a Schur-convex function can be obtained by f(t)  Schur-convex → Majority theory Remarks: The current tools in information theory don’t work. More sophisticated tools should be built to attack this problem. A new mathematical theory of information theory

Potential application: Interference channel  A challenge question: what is the application of GCMC?  Mathematical speaking, a beautiful result on a fundamental problem will be very useful  Potential Application Central limit theorem Capacity region of Gaussian broadcast channel Capacity region of Gaussian Multiple-Input Multiple-Output broadcast channel Uncertainty principle Where EPI works Gaussian interference channel: open since 1970s CM is considered to be much Where EPI fails more powerful than EPI

Remarks  If GCMC is true  A fundamental breakthrough in mathematical physics, information theory and any disciplines related to Gaussian distribution  A new expression for Fisher information  Derivatives are an invariant  Though ℎ(𝑌 + 𝑢𝑎) looks very messy, certain regularity exists  Application: Gaussian interference channel?  If GCMC is false  No Failure, as heat equation is a physical phenomenon  A lucky number (e.g. 2019) where Gaussian distribution fails. Painful!