Physics and/of Algorithms Michael (Misha) Chertkov Center for - PowerPoint PPT Presentation

Two Seemingly Unrelated Problems Physics of Algorithms: One Common Approach Some Technical Discussions (Results) Physics and/of Algorithms Michael (Misha) Chertkov Center for Nonlinear Studies & Theory Division, LANL and New Mexico Consortium September 16, 2011 Advanced Networks Colloquium at U of Maryland Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Physics of Algorithms: One Common Approach Some Technical Discussions (Results) Preliminary Remarks [on my strange path to the subjects] Phase transitions Quantum magnetism 1992 Statistical Hydrodynamics (passive scalar, turbulence) 1996 Discussed in This Talk 1999 Fiber Optics (noise, disorder) Mesoscopic 2004 Non-equilibrium Information Theory, CS Stat. Mech. Physics of Algorithms 2008 Optimization & Control Theory for Smart (Power) Grids Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Physics of Algorithms: One Common Approach Some Technical Discussions (Results) What to expect? [upfront mantra] From Algorithms to Physics and Back Inference (Reconstruction), Optimization & Learning, which are traditionally Computer/Information Science disciplines, allow Statistical Physics interpretations and benefit (Analysis & Algorithms) from using Physics ... and vice versa Interdisciplinary Stuff is Fun ... Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Physics of Algorithms: One Common Approach Some Technical Discussions (Results) Outline 1 Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Particle Tracking (Fluid Mechanics): Learning the Flow 2 Physics of Algorithms: One Common Approach Common Language (Graphical Models) & Common Questions Message Passing/ Belief Propagation ... and beyond ... (theory) 3 Some Technical Discussions (Results) Error Correction (Physics ⇒ Algorithms) Particle Tracking (Algorithms ⇒ Physics) Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Error Correction Scheme: Example of Additive White Gaussian Channel: � P ( x out | x in ) = p ( x out ; i | x in ; i ) i = bits p ( x | y ) ∼ exp( − s 2 ( x − y ) 2 / 2) Channel is noisy ”black box” with only statistical information available Encoding: use redundancy to redistribute damaging effect of the noise Decoding [Algorithm]: reconstruct most probable codeword by noisy (polluted) channel Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Low Density Parity Check Codes N bits, M checks, L = N − M information bits example: N = 10 , M = 5 , L = 5 2 L codewords of 2 N possible patterns Parity check: ˆ H v = c = 0 example:  1 1 1 1 0 1 1 0 0 0  0 0 1 1 1 1 1 1 0 0   ˆ H =  0 1 0 1 0 1 0 1 1 1    1 0 1 0 1 0 0 1 1 1   1 1 0 0 1 0 1 0 1 1 LDPC = graph (parity check matrix) is sparse Almost a tree! [Sparse Graph/Code] Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Decoding as Inference Statistical Inference ⇒ x ⇒ σ orig σ original corrupted possible data noisy channel data: statistical preimage σ orig ∈ C P ( x | σ ) log-likelihood inference σ ∈ C codeword magnetic field Maximum Likelihood Marginal Probability � arg max σ P ( σ | x ) arg max P ( x | σ ) σ i σ \ σ i Exhaustive search is generally expensive: complexity of the algorithm ∼ 2 N Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Decoding as Inference Statistical Inference ⇒ x ⇒ σ orig σ original corrupted possible data noisy channel data: statistical preimage σ orig ∈ C P ( x | σ ) log-likelihood inference σ ∈ C codeword magnetic field Maximum Likelihood Marginal Probability � arg max σ P ( σ | x ) arg max P ( x | σ ) σ i σ \ σ i Exhaustive search is generally expensive: complexity of the algorithm ∼ 2 N Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Decoding as Inference Statistical Inference ⇒ x ⇒ σ orig σ original corrupted possible data noisy channel data: statistical preimage σ orig ∈ C P ( x | σ ) log-likelihood inference σ ∈ C codeword magnetic field Maximum Likelihood Marginal Probability � arg max σ P ( σ | x ) arg max P ( x | σ ) σ i σ \ σ i Exhaustive search is generally expensive: complexity of the algorithm ∼ 2 N Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Decoding as Inference Statistical Inference ⇒ x ⇒ σ orig σ original corrupted possible data noisy channel data: statistical preimage σ orig ∈ C P ( x | σ ) log-likelihood inference σ ∈ C codeword magnetic field σ = ( σ 1 , · · · , σ N ) , N finite , σ i = ± 1 ( example ) Maximum Likelihood Marginal Probability � arg max σ P ( σ | x ) arg max P ( x | σ ) σ i σ \ σ i Exhaustive search is generally expensive: complexity of the algorithm ∼ 2 N Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Shannon Transition Existence of an efficient MESSAGE PASSING [belief propagation] decoding makes LDPC codes special! Phase Transition Ensemble of Codes [analysis & design] Thermodynamic limit but ... Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Error-Floor Ensembles of LDPC codes Old/bad codes BER vs SNR = measure of performance Error Rate Random Waterfall Finite size effects Waterfall ↔ Error-floor Optimized I Error-floor typically emerges due Optimized II Error floor to sub-optimality of decoding, i.e. due to unaccounted loops Signal-to-Noise Ratio Monte-Carlo is useless at FER � 10 − 8 T. Richardson ’03 (EF) Density evolution does not apply (to EF) Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Error-floor Challenges Ensembles of LDPC codes Old/bad codes Understanding the Error Floor (Inflection point, Asymptotics), Error Rate Need an efficient method to Random Waterfall analyze error-floor Improving Decoding Optimized I Constructing New Codes Optimized II Error floor Signal-to-Noise Ratio Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg

Two Seemingly Unrelated Problems Error Correction: Suboptimal decoding and Error-Floor Physics of Algorithms: One Common Approach Particle Tracking (Fluid Mechanics): Learning the Flow Some Technical Discussions (Results) Dance in Turbulence [movie] Learn the flow from tracking particles Michael (Misha) Chertkov – chertkov@lanl.gov https://sites.google.com/site/mchertkov/talks/phys-alg