What Physics has to do with Information Theory? Boltzmann, Gibbs - PowerPoint PPT Presentation

What Physics has to do with Information Theory? Boltzmann, Gibbs (19th century): statistical physics: macroscopic behavior from microscopic interactions E.T. Jaynes (1957): same principles can be derived from maximizing Shannon entropy H. Bethe (1935), R. Gallager (1963), J. Pearl (1982): similar ideas for exploiting sparsity in spin glasses, channel coding and AI late 90s – unified perspective: it’s all about inference computing avg. physical properties, decoding ECCs, learning in neural networks, denoising images, reconstructing signals . . . (M. Mezard, A. Montanari, F. Krzakala, R. Urbanke, D. Saad, Y. Kabashima, H. Nishimori, M. Opper, M. Jordan, M. Wainwright, . . . ) Andre Manoel (IF-USP, Brazil) Physics and Information 1 / 4

Example: decoding ECCs Q ( y | x ) − − − − − − − → y x transmitter noisy channel receiver 1 � � � � P ( x | y ) = Q ( y i | x i ) 1 ⊕ · · · ⊕ x i a p = 0 I x i a Z ( y ) a i � �� � � �� � prior likelihood single instance typical case compute marginals P ( x i | y ), compute free energy E y log Z ( y ), using belief propagation; using replica/cavity methods; assign symbol MAP estimate obtain avg. estimator performance x i ( y ) = arg max P ( x i | y ) ˆ � ε = E y I (ˆ x i ( y ) � = x i ) i Andre Manoel (IF-USP, Brazil) Physics and Information 2 / 4

What I’ve worked with in the recent past . . . Compressed sensing: sampling signals at sub-Nyquist rates y ∈ R M , x ∈ R N , F ∈ R M × N y = F x , Solve for sparse x when M ≪ N . Probabilistic approach: P ( x | y , F ) ∝ P ( y | F , x ) P 0 ( x ) Compute marginals using approximate message-passing (AMP) Been working on . . . variational approximations [arXiv:1402.1384] converging AMP algorithms [arXiv:1406.4311] (joint work w/ F. Krzakala, E. Tramel and L. Zdeborov´ a) Andre Manoel (IF-USP, Brazil) Physics and Information 3 / 4

. . . and what I’m working on right now From optimal source coding . . . min D ( x , y ) x s.t. x ∈ C e.g. using LDPC, x ∈ C ⇔ H x = 0, and BP for optimization . . . to information hiding min D ( x , y ) x ⇐ ⇒ embed message m in image y H ( κ ) x = m s.t. (joint work with Renato Vicente) Andre Manoel (IF-USP, Brazil) Physics and Information 4 / 4