Articial Neural Net w orks [Read Ch. 4] [Recommended - - PDF document
▶
SMART_READER_LITE
LIVE PREVIEW
Articial Neural Net w orks [Read Ch. 4] [Recommended - - PDF document
Dec 22, 2023 192 likes •512 views
Articial Neural Net w orks [Read Ch. 4] [Recommended exercises 4.1, 4.2, 4.5, 4.9, 4.11] Threshold units Gradien t descen t Multila y er net w orks Bac kpropagation Hidden la y er represen
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 2Connectionist Mo dels Consider h umans:
Neuron
switc hing time ~ :001 second
Num
b er
f
neurons ~ 10 10
Connections
p er neuron ~ 10 45
Scene
recognition time ~ :1 second
100
inference steps do esn't seem lik e enough ! m uc h parallel computation Prop erties
f
articial neural nets (ANN's):
Man
y neuron-lik e threshold switc hing units
Man
y w eigh ted in terconnections among units
Highly
parallel, distributed pro cess
Emphasis
n
tuning w eigh ts automatically 75 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 3When to Consider Neural Net w
rks
Input
is high-dimensional discrete
r
real-v alued (e.g. ra w sensor input)
Output
is discrete
r
real v alued
Output
is a v ector
f
v alues
P
ssibly
noisy data
F
rm
f
target function is unkno wn
Human
readabilit y
f
result is unimp
rtan
t Examples:
Sp
eec h phoneme recognition [W aib el]
Image
classication [Kanade, Baluja, Ro wley]
Financial
prediction 76 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 4AL VINN driv es 70 mph
n
high w a ys
Sharp Left Sharp Right
4 Hidden Units 30 Output Units 30x32 Sensor Input Retina
Straight Ahead
77 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 5P erceptron
w1 w2 wn w0 x1 x2 xn x0=1
. . .
Σ
AA
Σ wi xi
n i=0 1 if > 0
1 otherwise
{
=
Σ wi xi
n i=0
(x
1 ; : : : ; x n ) = 8 > > < > > : 1 if w + w 1 x 1 +
+
w n x n > 1
therwise.
Sometimes w e'll use simpler v ector notation:
(
~ x) = 8 > > < > > : 1 if ~ w
~
x > 1
therwise.
78 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 6Decision Surface
f
a P erceptron
x1 x2 + +
+
x1
x2
(a) (b)
+
+
Represen ts some useful functions
What
w eigh ts represen t g (x 1 ; x 2 ) = AN D (x 1 ; x 2 )? But some functions not represen table
e.g.,
not linearly separable
Therefore,
w e'll w an t net w
rks
f
these... 79 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 7P erceptron training rule w i w i + w i where w i =
(t
)x
i Where:
t
= c( ~ x ) is target v alue
is
p erceptron
utput
is
small constan t (e.g., .1) called le arning r ate 80 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 8P erceptron training rule Can pro v e it will con v erge
If
training data is linearly separable
and
sucien
tly small 81 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 9Gradien t Descen t T
understand,
consider simpler line ar unit, where
=
w + w 1 x 1 +
+
w n x n Let's learn w i 's that minimize the squared error E [ ~ w ]
1
2 X d2D (t d
d
) 2 Where D is set
f
training examples 82 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 10Gradien t Descen t
1
1 2
2
1
1 2 3 5 10 15 20 25 w0 w1 E[w]
Gradien t rE [ ~ w ]
2
6 4 @ E @ w ; @ E @ w 1 ;
@
E @ w n 3 7 5 T raining rule:
~
w =
rE
[ ~ w ] i.e., w i =
@
E @ w i 83 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 11Gradien t Descen t @ E @ w i = @ @ w i 1 2 X d (t d
d
) 2 = 1 2 X d @ @ w i (t d
d
) 2 = 1 2 X d 2(t d
d
) @ @ w i (t d
d
) = X d (t d
d
) @ @ w i (t d
~
w
~
x d ) @ E @ w i = X d (t d
d
)(x i;d ) 84 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 12Gradien t Descen t Gradient-Descent(tr aining exampl es;
)
Each tr aining example is a p air
f
the form h ~ x; ti, wher e ~ x is the ve ctor
f
input values, and t is the tar get
utput
value.
is
the le arning r ate (e.g., .05).
Initiali
ze eac h w i to some small random v alue
Un
til the termination condition is met, Do { Initiali ze eac h w i to zero. { F
r
eac h h ~ x ; ti in tr aining exampl es, Do
Input
the instance ~ x to the unit and compute the
utput
F
r
eac h linear unit w eigh t w i , Do w i w i +
(t
)x
i { F
r
eac h linear unit w eigh t w i , Do w i w i + w i 85 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 13Summary P erceptron training rule guaran teed to succeed if
T
raining examples are linearly separable
Sucien
tly small learning rate
Linear
unit training rule uses gradien t descen t
Guaran
teed to con v erge to h yp
thesis
with minim um squared error
Giv
en sucien tly small learning rate
Ev
en when training data con tains noise
Ev
en when training data not separable b y H 86 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 14Incremen tal (Sto c hastic) Gradien t Descen t Batc h mo de Gradien t Descen t: Do un til satised 1. Compute the gradien t rE D [ ~ w ] 2. ~ w ~ w
rE
D [ ~ w ] Incremen tal mo de Gradien t Descen t: Do un til satised
F
r
eac h training example d in D 1. Compute the gradien t rE d [ ~ w ] 2. ~ w ~ w
rE
d [ ~ w ] E D [ ~ w ]
1
2 X d2D (t d
d
) 2 E d [ ~ w ]
1
2 (t d
d
) 2 Incr emental Gr adient Desc ent can appro ximate Batch Gr adient Desc ent arbitrarily closely if
made
small enough 87 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 15Multila y er Net w
rks
f
Sigmoid Units
F1 F2 head hid who’d hood ... ...
88 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 16Sigmoid Unit
w1 w2 wn w0 x1 x2 xn x0 = 1
A A A A A
. . .
Σ
net = Σ wi xi
i=0 n
1 1 + e
net
= σ(net) =
(x)
is the sigmoid function 1 1 + e x Nice prop ert y: d (x) dx =
(x)(1
(x))
W e can deriv e gradien t decen t rules to train
One
sigmoid unit
Multilayer
networks
f
sigmoid units ! Bac kpropagation 89 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 17Error Gradien t for a Sigmoid Unit @ E @ w i = @ @ w i 1 2 X d2D (t d
d
) 2 = 1 2 X d @ @ w i (t d
d
) 2 = 1 2 X d 2(t d
d
) @ @ w i (t d
d
) = X d (t d
d
) B @
@
d
@ w i 1 C A =
X
d (t d
d
) @
d
@ net d @ net d @ w i But w e kno w: @
d
@ net d = @
(net
d ) @ net d =
d
(1
d
) @ net d @ w i = @ ( ~ w
~
x d ) @ w i = x i;d So: @ E @ w i =
X
d2D (t d
d
)o d (1
d
)x i;d 90 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 18Bac kpropagation Algorithm Initiali ze all w eigh ts to small random n um b ers. Un til satised, Do
F
r
eac h training example, Do 1. Input the training example to the net w
rk
and compute the net w
rk
utputs
2. F
r
eac h
utput
unit k
k
k
(1
k
)(t k
k
) 3. F
r
eac h hidden unit h
h
h
(1
h
) X k 2outputs w h;k
k
4. Up date eac h net w
rk
w eigh t w i;j w i;j w i;j + w i;j where w i;j =
j
x i;j 91 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 19More
n
Bac kpropagation
Gradien
t descen t
v
er en tire network w eigh t v ector
Easily
generalized to arbitrary directed graphs
Will
nd a lo cal, not necessarily global error minim um { In practice,
ften
w
rks
w ell (can run m ultiple times)
Often
include w eigh t momentum
w
i;j (n) =
j
x i;j + w i;j (n
1)
Minimizes
error
v
er tr aining examples { Will it generalize w ell to subsequen t examples?
T
raining can tak e thousands
f
iterations ! slo w!
Using
net w
rk
after training is v ery fast 92 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 20Learning Hidden La y er Represen tations
Inputs Outputs
A target function: Input Output 10000000 ! 10000000 01000000 ! 01000000 00100000 ! 00100000 00010000 ! 00010000 00001000 ! 00001000 00000100 ! 00000100 00000010 ! 00000010 00000001 ! 00000001 Can this b e learned?? 93 lecture slides for textb
k
Machine L e arning, T. Mitc hell, McGra w Hill, 1997
SLIDE 21Learning Hidden La y er Represen tations A net w
