Growth Rate of Spatially Coupled LDPC codes Workshop on Spatially - PowerPoint PPT Presentation

Growth Rate of Spatially Coupled LDPC codes 森 立平 Workshop on Spatially Coupled Codes and Related Topics at Tokyo Institute of Technology 2011/2/19

Contents 1. Factor graph, Bethe approximation and belief propagation 2. Relation between annealed free energy and belief propagation 3. Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon Here, growth rate is 1 G ( ω ) = lim N log E [ Z ( ω )] N →∞ Z ( ω ): the number of codewords of relative weight ω ∈ [0, 1]. 2 / 34

Factor graph, Bethe approximation and belief propagation 3 / 34

Factor graph Factor graph: bipartite graph which defines probability measure x ) = 1 � p ( x x f a ( x x x ∂ a ) Z a � � Z : = f a ( x x x ∂ a ), (partition function) x ∈X n x x a x ∂ a ) : X r a → R ≥ 0 f a ( x x 4 / 34

Gibbs free energy x ) = 1 � p ( x x f a ( x x x ∂ a ) Z a Approximation by simple distribution q of p which is defined by factor graph x ) log q ( x x ) x � D ( q � p ) = q ( x x p ( x x x ) x x x �� � � � = log Z − q ( x x ) log f a ( x x ∂ a ) + q ( x x ) log q ( x x ) x x x x x x x a x x x = : log Z + U ( q ) − H ( q ) = : log Z + F Gibbs ( q ) U ( p ): internal energy H ( p ): entropy F Gibbs ( p ): Gibbs free energy 5 / 34

Mean field approximation and Bethe approximation Mean field approximation � q ( x x x ) = b i ( x i ) i Degree of freedom is reduced from q n to nq Bethe approximation � a b a ( x x ∂ a ) x q ( x x ) = x � i b i ( x i ) d i − 1 d i : degree of variable node i When factor graph is tree, Bethe approximation can be exact 6 / 34

Bethe free energy �� � � U ( q ) = − q ( x x x ) log f a ( x x x ∂ a ) x x x a � � b a ( x x x ∂ a ) log f a ( x x x ∂ a ) = : U Bethe ( { b a } ) ≈− a x x x ∂ a � a b a ( x x x ∂ a ) b ( x x x ) ≈ � x ) d i − 1 i b i ( x x � H ( b ) = − b ( x ) log b ( x ) x x x � a b a ( x x ∂ a ) x � b ( x ) log ≈− � x ) d i − 1 i b i ( x x x x x � � � � = − b a ( x x ∂ a ) log b a ( x x ∂ a ) + ( d i − 1) b i ( x i ) log b i ( x i ) x x x a x x ∂ a i i = : H Bethe ( { b i } , { b a } ) 7 / 34

Minimization of Bethe free energy F Bethe ( { b i } , { b a } ) : = U Bethe ( { b a } ) − H Bethe ( { b i } , { b a } ) minimize : F Bethe ( { b i } , { b a } ) subject to : b i ( x i ) ≥ 0, ∀ i b a ( x x x ∂ a ) ≥ 0, ∀ a � b i ( x i ) = 1 i � b a ( x x ∂ a ) = 1 x a � b a ( x x ∂ a ) = b i ( x i ), ∀ a , ∀ i ∈ ∂ a x x x x ∂ a \ x i 8 / 34

Stationary point of Lagrangian of Bethe free energy [Yedidia, Freeman, and Weiss 2005] �� � �� � � � L : = F Bethe ( { b i } , { b a } ) + b a ( x x x ∂ a ) − 1 + b i ( x ) − 1 γ a γ i x a x x ∂ a x i   � � � � + λ ai ( x i )  b i ( x i ) − b a ( x x ∂ a ) x  a x i i ∈ ∂ a x ∂ a \ x i x x Stationary points of Lagrangian is fixed points of BP � b a ( x x ∂ a ) ∝ f a ( x x ∂ a ) m i → a ( x i ) x x i ∈ ∂ a � b i ( x i ) ∝ m a → i ( x i ) i ∈ ∂ a where � m i → a ( x i ) ∝ m c → i ( x i ) c ∈ ∂ i \ a � � m a → i ( x i ) ∝ f a ( x x x ∂ a ) m j → a ( x j ) x ∂ a \ x i x j ∈ ∂ a \ i x 9 / 34

Relation between annealed free energy and belief propagation 10 / 34

Random regular factor graph ensemble Factor graph: bipartite graph which defines probability measure x ) = 1 � µ ( x x f a ( x x ∂ a ) x Z a � � Z : = f a ( x x x ∂ a ), (partition function) x ∈X n x x a Random ( l , r )-regular factor graph ensemble: l : degree of variable nodes, r : degree of factor nodes Random ensemble of factor graphs 11 / 34

Annealed free energy Factor graph: bipartite graph which defines probability measure x ) = 1 � µ ( x x f a ( x x ∂ a ) x Z a � � Z : = f a ( x x x ∂ a ), (partition function) x ∈X n x x a Random ( l , r )-regular factor graph ensemble: l : degree of variable nodes, r : degree of factor nodes Random ensemble of factor graphs (Quenched) free energy: 1 lim N E [log Z ] N →∞ Annealed free energy: 1 lim N log E [ Z ] N →∞ 12 / 34

Contribution to partition function of particular types { v x } x ∈X : the number of variable nodes of value x ∈ X is v x x ∈ X r is u x { u x x } x x ∈X r : the number of factor nodes of value x x x x x x � � Z = f ( x x x ∂ a ) a x ∈X N x x � � x ) u x = N ( { v } , { u } ) f ( x x x . x x ∈X r x { v } , { u } x l �� � �� x ∈X ( v x l )! N r N E [ N ( { v } , { u } )] = . { v x } x ∈X { u x x } x ( Nl )! x ∈X r x x 1 lim N log E [ Z ( { ν } , { µ } )] N →∞ = l r H ( { µ } ) − ( l − 1) H ( { ν } ) + l � µ ( x x x ) log f ( x x x ). r x ∈X r x x 13 / 34

Annealed free energy of fixed type and Bethe free energy � � F Bethe ( { b i } , { b a } ) = − b a ( x x x ∂ a ) log f a ( x x ∂ a ) x x a x x ∂ a � � � � + b a ( x x ∂ a ) log b a ( x x ∂ a ) − ( d i − 1) b i ( x i ) log b i ( x i ) x x x a x x ∂ a i i 1 lim N log E [ Z ( { ν } , { µ } )] N →∞ = l x ) + l � µ ( x x x ) log f ( x x r H ( { µ } ) − ( l − 1) H ( { ν } ). r x ∈X r x x 14 / 34

Maximization of the exponents of contributions r H ( { µ } ) − ( l − 1) H ( { ν } ) + l l � maximize : µ ( x x ) log f ( x x x ) x r x ∈X r x x subject to : ν ( x ) ≥ 0, ∀ x ∈ X x ∈ X r µ ( x x x ) ≥ 0, ∀ x x � ν ( x ) = 1 x ∈X � µ ( x x ) = 1 x x ∈X r x x r 1 � � µ ( x x ) = ν ( z ), x ∀ z ∈ X r k =1 x x \ x k x x k = z 15 / 34

The stationary condition The stationary condition is ν ( x ) ∝ m f → v ( x ) l r � µ ( x x ) ∝ f ( x x ) m v → f ( x i ) x x i =1 where m v → f ( x ) ∝ m f → v ( x ) l − 1 r � � � m f → v ( x ) ∝ f ( x x x ) m v → f ( x j ). k =1 j � = k x x \ x k x x k = x x ∈ X r If f ( x x x ) is invariant under any permutation of x x � � m f → v ( x ) ∝ f ( x x x ) m v → f ( x j ). j � =1 x x x \ x 1 x 1 = x 16 / 34

Annealed free energy Theorem 1. � l � 1 lim N log E [ Z ] = max r log Z f + log Z v − l log Z fv . N →∞ ( m f → v , m v → f ) ∈ S where S denotes the set of saddle points, and where � m f → v ( x ) l Z v : = x r � � Z f : = f ( x x ) x m v → f ( x i ) x x x i =1 � Z fv : = m f → v ( x ) m v → f ( x ). x 17 / 34

Number of solutions If r � � f ( x x x ) k =1 x x \ x k x x k = x is constant among all x ∈ X , the uniform message m f → v ( x ) 、 m v → f ( x ) is a saddle point. The contribution of the uniform message is � N f � 1 N log E [ Z ( ν , µ )] = log q + l lim r log (design rate) q r N →∞ where q : = |X| 、 N f : = � x f ( x x ). x x x For CSP i.e., f ( x x ) ∈ { 0, 1 } , the expected number of solutions is about x � l r N � N f q N . q r This intuitively means all constraints are independent. 18 / 34

Contribution to partition function of fixed variable type � Z ( { ν } ) : = Z ( { ν } , { µ } ) { µ } 1 lim N log E [ Z ( { ν } )] N →∞ � � r H ( { µ } ) − ( l − 1) H ( { ν } ) + l l � = sup µ ( x x ) log f ( x x ) x x r { µ } x ∈X r x x where { µ } satisfies x ∈ X r µ ( x x ) ≥ 0, ∀ x x x � µ ( x x x ) = 1 x ∈X r x x r 1 � � µ ( x x ) = ν ( z ), x ∀ z ∈ X r k =1 x \ x k x x x k = z Convex optimization problem with linear constraints. 19 / 34

The stationary condition The stationary condition is r � µ ( x x ) ∝ f ( x x ) m v → f ( x i ) x x i =1 where ν ( x ) ∝ h ( x ) m f → v ( x ) l m v → f ( x ) ∝ h ( x ) m f → v ( x ) l − 1 r � � � m f → v ( x ) ∝ f ( x x x ) m v → f ( x j ). k =1 j � = k x x x \ x k x k = x x ∈ X r If f ( x x ) is invariant under any permutation of x x x � � m f → v ( x ) ∝ f ( x x ) m v → f ( x j ). x j � =1 x \ x 1 x x x 1 = x 20 / 34

Growth rate of contribution to partition function of fixed variable type Theorem 2. 1 lim N log E [ Z ( { ν } )] N →∞ � � l � = max r log Z f + log Z v − l log Z fv − ν ( x ) log h ( x ) ( m f → v , m v → f ) ∈ S x where S denotes the set of saddle points, and where � h ( x ) m f → v ( x ) l Z v : = x r � � Z f : = f ( x x ) x m v → f ( x i ) x x x i =1 � Z fv : = m f → v ( x ) m v → f ( x ). x 21 / 34

Growth rate of regular LDPC codes � � l � � l r log 1 + z r � 1 + y � 1 − y G ( ω ) = l e h + e − h + log 2 2 2 − l log 1 + yz − ω ′ h 2 where ω ′ : = 1 − 2 ω and ω ′ = tanh( h + l tanh − 1 ( y )) y = z r − 1 z = tanh( h + ( l − 1) tanh − 1 ( y )). This result can be easily understood from correspondings ω ′ = ν (0) − ν (1) y = m f → v (0) − m f → v (1) z = m v → f (0) − m v → f (1) h ( x ) = e ( − 1) x h 22 / 34

Growth rate of regular LDPC codes 0.4 0 (5,10) 0.3 (6,10) (7,10) (8,10) 0.2 (9,10) (10,10) 0.1 (11,10) 0 G -0.1 -0.2 -0.3 -0.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 w 23 / 34

Growth rate of binary CSP 0.7 k=1 k=2 k=3 0.6 0.5 0.4 G 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 w 24 / 34