9.54 Review Supervised learning + Invariant Learning Shimon Ullman - PowerPoint PPT Presentation

9.54 Review Supervised learning + Invariant Learning Shimon Ullman + Tomaso Poggio Danny Harari + Daneil Zysman + Darren Seibert 9.54, fall semester 2014

Thus Y − MX = 0 min || Y − MX || 2 More in general look for M such that if M = w T r V ( w ) = 2( Y � w T X ) X T = 0 yields Y X T = w T XX T and w T = Y X T ( XX T ) − 1 min w || Y − w T X || 2 + λ || w || 2 Now look for w such that r V ( w ) = 2( Y � w T X ) X T + 2 λ w T = 0 yields Y X T = w T XX T + λ w T and w T = Y X T ( XX T + λ I ) − 1

Example: representer theorem in the linear case Math Some simple linear algebra shows that w T = Y X T ( XX T ) − 1 = Y ( X T X ) − 1 X T = CX T since X T ( XX T ) − 1 = ( X T X ) − 1 X T Then n f ( x ) = w T x = CX T x = X c i x T i x i We can compute C n or w n depending whether n ≤ p . The above result is the most basic form of the Representer Theorem.

M-theory: exploring a new hypothesis The main computational goal of the feedforward ventral stream hierarchy — and of vision — is to compute a representation for each incoming image which is invariant to transformations previously experienced in the visual environment. Remarks: • A theorem shows that invariant representations may reduce by orders of magnitude the sample complexity of a classifier at the top of the hierarchy • Empirical evidence also supports the claim • Intuitions include the decrease in appearance that invariance afford

Transformation example: affine group g I The action of a group transformation on an image is defined as: gI ( ! x ) = I ( g − 1 ! x ) In the case of affine group: ! ! gI ( ! x ) = I ( A − 1 ! x − b ), A ∈ GL (2), b ∈ R 2

Transformation example: affine group The action of a group transformation on an image is defined as: In the case of affine group:

Our basic machine: a HW module (dot products and histograms for an image in a receptive field window) • The cumulative histogram (empirical cdf) can be be computed as | G | σ ( I , g i t k + n Δ ) 1 ∑ µ n k ( I ) = ... | G | i = 1 n Δ • This maps directly into a set of simple cells with threshold • …and a complex cell indexed by n and k summating the simple cells

Dendrites of a complex cells as simple cells … Active properties in the dendrites of the complex cell

Our basic machine: a HW module • The (empirical) first moment — e.g. the average — is | G | m 1 = 1 ∑ I , g i t k | G | i = 1 ... • The (empirical) second moment is | G | m 2 = 1 ∑ ( I , g i t k ) 2 | G | i = 1 • …and the infinite order moment is m ∞ = max i I , g i t k

Dendrites of a complex cells as simple cells … Active properties in the dendrites of the complex cell