Neural Networks Hopfield Nets and Auto Associators Fall 2017 1 - PowerPoint PPT Presentation

Neural Networks Hopfield Nets and Auto Associators Fall 2017 1

Story so far • Neural networks for computation • All feedforward structures • But what about.. 2

Loopy network Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 −1 𝑗𝑔 𝑨 ≤ 0 𝑘≠𝑗 The output of a neuron affects the input to the neuron • Each neuron is a perceptron with +1/-1 output • Every neuron receives input from every other neuron • Every neuron outputs signals to every other neuron 3

Loopy network Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 −1 𝑗𝑔 𝑨 ≤ 0 𝑘≠𝑗 A symmetric network: 𝑥 𝑗𝑘 = 𝑥 𝑘𝑗 • Each neuron is a perceptron with +1/-1 output • Every neuron receives input from every other neuron • Every neuron outputs signals to every other neuron 4

Hopfield Net Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 −1 𝑗𝑔 𝑨 ≤ 0 𝑘≠𝑗 A symmetric network: 𝑥 𝑗𝑘 = 𝑥 𝑘𝑗 • Each neuron is a perceptron with +1/-1 output • Every neuron receives input from every other neuron • Every neuron outputs signals to every other neuron 5

Loopy network Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 −1 𝑗𝑔 𝑨 ≤ 0 𝑘≠𝑗 A neuron “flips” if weighted sum of other neuron’s outputs is of the opposite sign But this may cause other neurons to flip! • Each neuron is a perceptron with a +1/-1 output • Every neuron receives input from every other neuron • Every neuron outputs signals to every other neuron 6

Loopy network 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 −1 𝑗𝑔 𝑨 ≤ 0 • At each time each neuron receives a “field” σ 𝑘≠𝑗 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 • If the sign of the field matches its own sign, it does not respond • If the sign of the field opposes its own sign, it “flips” to match the sign of the field 7

Example • Red edges are -1, blue edges are +1 • Yellow nodes are +1, black nodes are -1 8

Example • Red edges are -1, blue edges are +1 • Yellow nodes are +1, black nodes are -1 9

Example • Red edges are -1, blue edges are +1 • Yellow nodes are +1, black nodes are -1 10

Example • Red edges are -1, blue edges are +1 • Yellow nodes are +1, black nodes are -1 11

Loopy network • If the sign of the field at any neuron opposes its own sign, it “flips” to match the field – Which will change the field at other nodes • Which may then flip – Which may cause other neurons including the first one to flip… » And so on… 12

20 evolutions of a loopy net A neuron “flips” if Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 weighted sum of other −1 𝑗𝑔 𝑨 ≤ 0 neuron’s outputs is of the opposite sign 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 But this may cause 𝑘≠𝑗 other neurons to flip! • All neurons which do not “align” with the local field “flip” 13

120 evolutions of a loopy net • All neurons which do not “align” with the local field “flip” 14

Loopy network • If the sign of the field at any neuron opposes its own sign, it “flips” to match the field – Which will change the field at other nodes • Which may then flip – Which may cause other neurons including the first one to flip… • Will this behavior continue for ever?? 15

Loopy network 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 −1 𝑗𝑔 𝑨 ≤ 0 − be the output of the i- th neuron just before it responds to the • Let 𝑧 𝑗 current field + be the output of the i- th neuron just after it responds to the current • Let 𝑧 𝑗 field + = 𝑧 𝑗 − = 𝑡𝑗𝑕𝑜 σ 𝑘≠𝑗 𝑥 − • If 𝑧 𝑗 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 , then 𝑧 𝑗 – If the sign of the field matches its own sign, it does not flip + − 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 − 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 = 0 𝑘≠𝑗 𝑘≠𝑗 16

Loopy network 𝑧 𝑗 = Θ ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 Θ 𝑨 = ቊ+1 𝑗𝑔 𝑨 > 0 −1 𝑗𝑔 𝑨 ≤ 0 − ≠ 𝑡𝑗𝑕𝑜 σ 𝑘≠𝑗 𝑥 + = −𝑧 𝑗 − • If 𝑧 𝑗 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 , then 𝑧 𝑗 + − + 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 − 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 = 2𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 𝑘≠𝑗 𝑘≠𝑗 – This term is always positive! • Every flip of a neuron is guaranteed to locally increase 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 17

Globally • Consider the following sum across all nodes 𝐸 𝑧 1 , 𝑧 2 , … , 𝑧 𝑂 = ෍ 𝑧 𝑗 ෍ 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑗 𝑘<𝑗 = ෍ 𝑥 𝑗𝑘 𝑧 𝑗 𝑧 𝑘 + ෍ 𝑐 𝑗 𝑧 𝑗 𝑗,𝑘<𝑗 𝑗 – Definition same as earlier, but avoids double counting and assumes 𝑥 𝑗𝑗 = 0 • For any unit 𝑙 that “flips” because of the local field + , … , 𝑧 𝑂 − 𝐸 𝑧 1 , … , 𝑧 𝑙 − , … , 𝑧 𝑂 ∆𝐸 𝑧 𝑙 = 𝐸 𝑧 1 , … , 𝑧 𝑙 18

Upon flipping a single unit + , … , 𝑧 𝑂 − 𝐸 𝑧 1 , … , 𝑧 𝑙 − , … , 𝑧 𝑂 ∆𝐸 𝑧 𝑙 = 𝐸 𝑧 1 , … , 𝑧 𝑙 • Expanding + − 𝑧 𝑙 − ෍ + − 𝑧 𝑙 − 𝑐 𝑙 ∆𝐸 𝑧 𝑙 = 𝑧 𝑙 𝑥 𝑘𝑙 𝑧 𝑘 + 𝑧 𝑙 𝑘≠𝑗 – All other terms that do not include 𝑧 𝑙 cancel out • This is always positive! • Every flip of a unit results in an increase in 𝐸 19

Hopfield Net • Flipping a unit will result in an increase (non-decrease) of 𝐸 = ෍ 𝑥 𝑗𝑘 𝑧 𝑗 𝑧 𝑘 + ෍ 𝑐 𝑗 𝑧 𝑗 𝑗,𝑘<𝑗 𝑗 • 𝐸 is bounded 𝐸 𝑛𝑏𝑦 = ෍ 𝑥 𝑗𝑘 + ෍ 𝑐 𝑗 𝑗,𝑘<𝑗 𝑗 • The minimum increment of 𝐸 in a flip is ∆𝐸 𝑛𝑗𝑜 = 𝑗, {𝑧 𝑗 , 𝑗=1..𝑂} 2 ෍ min 𝑥 𝑘𝑗 𝑧 𝑘 + 𝑐 𝑗 𝑘≠𝑗 • Any sequence of flips must converge in a finite number of steps 20

The Energy of a Hopfield Net • Define the Energy of the network as 𝐹 = − ෍ 𝑥 𝑗𝑘 𝑧 𝑗 𝑧 𝑘 − ෍ 𝑐 𝑗 𝑧 𝑗 𝑗,𝑘<𝑗 𝑗 – Just the negative of 𝐸 • The evolution of a Hopfield network constantly decreases its energy • Where did this “energy” concept suddenly sprout from? 21

Analogy: Spin Glasses • Magnetic diploes • Each dipole tries to align itself to the local field – In doing so it may flip • This will change fields at other dipoles – Which may flip • Which changes the field at the current dipole… 22

Analogy: Spin Glasses Total field at current dipole: 𝑠𝑦 𝑘 𝑔 𝑞 𝑗 = ෍ 2 + 𝑐 𝑗 𝑞 𝑗 − 𝑞 𝑘 𝑘≠𝑗 intrinsic external • 𝑞 𝑗 is vector position of 𝑗 -th dipole • The field at any dipole is the sum of the field contributions of all other dipoles • The contribution of a dipole to the field at any point falls off inversely with square of distance 23

Analogy: Spin Glasses Total field at current dipole: 𝑠𝑦 𝑘 𝑔 𝑞 𝑗 = ෍ 2 + 𝑐 𝑗 𝑞 𝑗 − 𝑞 𝑘 𝑘≠𝑗 Response of current diplose 𝑦 𝑗 = ൝𝑦 𝑗 𝑗𝑔 𝑡𝑗𝑕𝑜 𝑦 𝑗 𝑔 𝑞 𝑗 = 1 −𝑦 𝑗 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 • A Dipole flips if it is misaligned with the field in its location 24

Analogy: Spin Glasses Total field at current dipole: 𝑠𝑦 𝑘 𝑔 𝑞 𝑗 = ෍ 2 + 𝑐 𝑗 𝑞 𝑗 − 𝑞 𝑘 𝑘≠𝑗 Response of current diplose 𝑦 𝑗 = ൝𝑦 𝑗 𝑗𝑔 𝑡𝑗𝑕𝑜 𝑦 𝑗 𝑔 𝑞 𝑗 = 1 −𝑦 𝑗 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 • Dipoles will keep flipping – A flipped dipole changes the field at other dipoles • Some of which will flip – Which will change the field at the current dipole • Which may flip – Etc.. 25

Analogy: Spin Glasses Total field at current dipole: 𝑠𝑦 𝑘 𝑔 𝑞 𝑗 = ෍ 2 + 𝑐 𝑗 𝑞 𝑗 − 𝑞 𝑘 𝑘≠𝑗 Response of current diplose • When will it stop??? 𝑦 𝑗 = ൝ 𝑦 𝑗 𝑗𝑔 𝑡𝑗𝑕𝑜 𝑦 𝑗 𝑔 𝑞 𝑗 = 1 −𝑦 𝑗 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 26

Analogy: Spin Glasses Total field at current dipole: 𝑠𝑦 𝑘 𝑔 𝑞 𝑗 = ෍ 2 + 𝑐 𝑗 𝑞 𝑗 − 𝑞 𝑘 𝑘≠𝑗 Response of current diplose 𝑦 𝑗 = ൝𝑦 𝑗 𝑗𝑔 𝑡𝑗𝑕𝑜 𝑦 𝑗 𝑔 𝑞 𝑗 = 1 −𝑦 𝑗 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 • The total potential energy of the system 𝐹 = 𝐷 − 1 𝑠𝑦 𝑗 𝑦 𝑘 2 ෍ 𝑦 𝑗 𝑔 𝑞 𝑗 = 𝐷 − ෍ ෍ 2 − ෍ 𝑐 𝑗 𝑦 𝑘 𝑞 𝑗 − 𝑞 𝑘 𝑗 𝑗 𝑘>𝑗 𝑗 • The system evolves to minimize the PE – Dipoles stop flipping if any flips result in increase of PE 27

Spin Glasses PE state • The system stops at one of its stable configurations – Where PE is a local minimum • Any small jitter from this stable configuration returns it to the stable configuration – I.e. the system remembers its stable state and returns to it 28