Dense Associative Memories and Deep Learning Dmitry Krotov IBM - PowerPoint PPT Presentation

Dense Associative Memories and Deep Learning Dmitry Krotov IBM Research MIT-IBM Watson AI Lab Institute for Advanced Study

Learning Mechanisms Architectures

What is associative memory? energy landscape ξ 1 ξ 2 ξ 3 ξ 4 memories

Standard Dense Associative Memory Associative Memory N X E = − σ i T ij σ j K X ξ µ i ξ µ K ⇣ N i,j =1 T ij = ⌘ n j X X ξ µ E = − i σ i µ =1 µ =1 i =1 -dynamical variables σ i n ≥ 2 ξ µ -memorized patterns i power of the N -number of neurons interaction vertex K -number of memories K ⇣ N ⌘ 2 X X ξ µ E = − i σ i µ =1 i =1 K max ≈ 0 . 14 N K max ≈ α n N n − 1

 K ✓ ⌘◆� ⇣ ⌘ ⇣ σ ( t +1) j σ ( t ) j σ ( t ) X X X ξ µ ξ µ � ξ µ ξ µ = Sign F i + � F i + i j j µ =1 j 6 = i j 6 = i h ξ µ i i = 0 h ξ µ j i = δ µ ν δ ij i ξ ν

Pattern recognition with DAM classification visible neurons neurons 28 v i = v i x α or c α 784 10 28

 K ✓ ⌘◆� ⇣ ⌘ ⇣ σ ( t +1) j σ ( t ) j σ ( t ) X X X ξ µ ξ µ � ξ µ ξ µ = Sign F i + � F i + i j j µ =1 j 6 = i j 6 = i K N N  ✓ ⌘◆� ⇣ ⌘ ⇣ X X X ξ µ X X ξ µ − ξ µ ξ µ ξ µ ξ µ c α = g α x α + γ x γ + α x α + γ x γ + β F i v i − F i v i µ =1 i =1 i =1 γ 6 = α γ 6 = α output c α . The update g ( x ) = tanh( x )

training random memories constructed memory ξ µ i ∈ N (0 , 0 . 1) vectors MNIST Dataset

Main question: What kind of representation of the data has the neural network learned?

Features vs. prototypes in psychology and neuroscience Feature-matching theory Prototype theory Electrical signal from brain Visual area of brain Recording electrode Stimulus Hubel,Wiesel, 1959 Solso, McCarthy,1981 training set Wallis, et al., Journal of Vision,2008

Feature to prototype transition power of the interaction vertex n = 3 n = 20 n = 2 n = 30 1 256 0 . 5 192 0 128 − 0 . 5 64 − 1 feature detectors prototype detectors

Feature to prototype transition power of the interaction vertex n = 3 n = 20 n = 2 n = 30 1 256 0 . 5 192 0 128 − 0 . 5 64 − 1 1 . 44% 1 . 51% 1 . 61% 1 . 80% 1 . 6% Simard, Steinkraus, Platt, 2003

Duality with feed-forward nets c α ⇣ K ⌘ c α X v i ξ µ c α = g α h µ µ =1 h µ ⇣ N v i x α ⌘ X ξ µ h µ = f i v i i =1 K ⇣ N 10 v i ⌘ X X X ξ µ ξ µ E = − F i v i + α c α µ =1 i =1 α =1 Duality rule: f ( x ) = F 0 ( x ) activation energy function function

Commonly used activation functions n = 2 n standard DAM Hopfield net f ( x ) = ReP n − 1 f ( x ) = ReLU x x

Question: Are there any tasks for which models with higher order interactions perform better than models with quadratic interactions?

Adversarial Inputs n=2 2 3 v i → v i − ∂ C ∂ v i

Adversarial Deformations in DAM 10 C 1st log( C α ) A A 0 C 2nd A A -10 decision boundary -20 10 20 30 40 50 60 70 80 number of image updates n=2 n=3 n=20 n=30 8 3 8 3 3 3 3 9 5 8 8 8

Question: Can we use Dense Associative Memories for classification of high resolution images?

VGG16 coupled to DAM �������������������������������� ��������������������������� ���� ���

Adversarial Inputs in the Image Domain ����������������������������

Input transfer classified by n=2 classified by n=8 made with n=2 ������������������� classified by n=2 classified by n=8 made with n=8 �������������������

Error rate of misclassification Classify n=2 n=8 Generate n=2 100% 32% n=8 57% 100%

n=30 37.6% 48.3% 56.9% 98.8% generate n=20 45.3% 63.7% 98.9% 5.77% n=3 33.9% 99% 8.71% 3.32% 98.9% 50.7% 9.07% 3.44% n=2 n=2 n=3 n=20 n=30 test

Results on ImageNet Accuracy: 69%

ImageNet errors police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria bell cote, bell cot

Large Capacity Physics Dense Associative Memories K ⇣ N ⌘ n X X ξ µ E = − i σ i µ =1 i =1 Psychology Computer Neuroscience Science No Adversarial Feature to Prototype Problems Transition