Finding Structure in Time Finding Structure in Time By Jonathan Hall - - PowerPoint PPT Presentation

finding structure in time finding structure in time
SMART_READER_LITE
LIVE PREVIEW

Finding Structure in Time Finding Structure in Time By Jonathan Hall - - PowerPoint PPT Presentation

Finding Structure in Time Finding Structure in Time By Jonathan Hall Author: Jeffrey L. Elman Author: Jeffrey L. Elman Contents Contents Problem Problem Algorithm Results l Conclusion Problem Problem Speech research Speech


slide-1
SLIDE 1

Finding Structure in Time Finding Structure in Time

By Jonathan Hall Author: Jeffrey L. Elman Author: Jeffrey L. Elman

slide-2
SLIDE 2

Contents Contents

  • Problem

Problem

  • Algorithm

l

  • Results
  • Conclusion
slide-3
SLIDE 3

Problem Problem

  • Speech research

Speech research

  • Usual MLP takes all input at once and returns
  • utput all at once
  • utput all at once
  • Speech needs temporal ordering
slide-4
SLIDE 4

Past solutions to problem Past solutions to problem

  • Usual approach: sequential representation of

Usua app oac : seque t a ep ese tat o o events

– [0 1 1 1 0 0 0 0 0] – [0 0 0 1 1 1 0 0 0]

  • Biological Implications

– How to express parallel computations in terms of human brain?

  • Not all input vectors same length
  • Not all input vectors same length

– Example: translating user speech into text

  • Applications: prediction problems
  • Applications: prediction problems
slide-5
SLIDE 5

Solution to Problem Solution to Problem

  • Variable length input

Variable length input

  • Analyze batches of events

i

  • Dynamic system
  • Remember past events
slide-6
SLIDE 6

Algorithm Algorithm

  • Solution: give network memory with a

Solution: give network memory with a recurrent network

slide-7
SLIDE 7

Recurrent Networks Overview Recurrent Networks Overview

  • Has a directed cycle

Has a directed cycle

  • Hopfield is a symmetric recurrent network
  • Good for: learning grammar speech recognition and

Good for: learning grammar, speech recognition, and music composition

slide-8
SLIDE 8

Author’s Network Author s Network

  • Why did author use network on left and not

Why did author use network on left and not the one on the right?

slide-9
SLIDE 9

Results Contents Results Contents

  • XOR

XOR

  • Letter Sequence

d

  • Words
  • Sentences
slide-10
SLIDE 10

XOR XOR

  • Problem setup:

Problem setup:

– Input: 2 random bits, 3rd bit is XOR of previous 2 bits bits – Goal: predict next bit – Example Input: 1 0 1 0 0 0 0 1 1 1 1 0 1 0 1 – Example Input: 1 0 1 0 0 0 0 1 1 1 1 0 1 0 1…

slide-11
SLIDE 11

XOR Results XOR Results

slide-12
SLIDE 12

Letter Sequence Letter Sequence

  • Problem setup:

Problem setup:

– Input: Random consonants (b,d,g), then random consonants replaced with: b‐>ba d‐>dii g‐>guuu consonants replaced with: b >ba, d >dii, g >guuu – Each letter given unique 1x6 vector – Goal: predict next letter – Goal: predict next letter

slide-13
SLIDE 13

Letter Sequence Results Letter Sequence Results

slide-14
SLIDE 14

Words Words

  • Problem Setup:

Problem Setup:

– Input: String of sentences with no breaks between words or sentences words or sentences – Goal: predict next letter of sequence

slide-15
SLIDE 15

Words Results Words Results

slide-16
SLIDE 16

Sentence Setup Sentence Setup

  • A large number of simple sentences were

A large number of simple sentences were randomly produced

  • Each word was vectorized
  • Each word was vectorized
  • No breaks in between sentences
  • Goal: predict the next word
slide-17
SLIDE 17

Sentences Sentences

slide-18
SLIDE 18

Sentences: quality measurement Sentences: quality measurement

  • Can’t use word‐by‐word RSS

Can t use word by word RSS

– Error: 0.88

  • Solution use RSS for categories of words
  • Solution: use RSS for categories of words

– Error: 0.053

  • How did the network do this?
slide-19
SLIDE 19

Sentence Classification Sentence Classification

slide-20
SLIDE 20

Conclusion Conclusion

  • Some problems are different when expressed

Some problems are different when expressed as temporal events.

  • RSS can be used to analyze the temporal

RSS can be used to analyze the temporal structure.

  • Length of sequential dependencies doesn’t

Length of sequential dependencies doesn t always worsen performance.

  • Representation of time and memory is task‐

Representation of time and memory is task dependant.

  • Representations can be structured.

Representations can be structured.

slide-21
SLIDE 21

The End