Learning in One-Layer Networks Psych 209 January 9, 2020
Input-output mapping Simplest model of learning: input-output mapping
Input-output mapping Pattern associator a1 b1 a2 b2 . . . . . . an bn Input stimulus Input pattern Output pattern Output behavior
Input-output mapping How can we learn input-output associations? We know how to store (key, value) pairs on a computer a1 b1 a2 b2 . . hash address . . . . an bn Output pattern Input pattern
Input-output mapping But location-based memory is brittle a1 b1 a2 b2 . . hash address . . . . an bn a1’ z1 a2’ z2 . di ff erent . hash . address . . . an’ zn Similar input Totally di ff erent patterns output pattern
Pattern Associator Content-addressable memory: keys directly “produce” values without location lookup Input pattern Output pattern neuron (cell body) connection (axon) Desirable properties Fully connected
Forward Pass Input pattern Output pattern Input 0 w 00 w 01 Input 1 Output 0 w 02 Input 2 Output 1 w 03 Input 3
<latexit sha1_base64="lLa6sjSOQSA6ZvKbViClURecGQ=">ACAHicbZDNSsNAFIUn9a/Wv6gbwc1gEVyVRAR1IRTduKxgbaEN4WY6aedTMLMRCmhbnwVNy5U3PoY7nwbp20W2npg4OPce7lzT5BwprTjfFuFhcWl5ZXiamltfWNzy97euVNxKgmtk5jHshmAopwJWtdMc9pMJIUo4LQRDK7G9cY9lYrF4lYPE+pF0BUsZAS0sXx7D3yGL3BbpZHfxw9+xvojDAZ9u+xUnInwPLg5lFGum9/tTsxSMqNOGgVMt1Eu1lIDUjnI5K7VTRBMgAurRlUEBElZdNLhjhQ+N0cBhL84TGE/f3RAaRUsMoMJ0R6J6arY3N/2qtVIdnXsZEkmoqyHRmHKsYzyOA3eYpETzoQEgkpm/YtIDCUSb0EomBHf25HmoH1fOK+7NSbl6madRPvoAB0hF52iKrpGNVRHBD2iZ/SK3qwn68V6tz6mrQUrn9lFf2R9/gBTDZW6</latexit> <latexit sha1_base64="lLa6sjSOQSA6ZvKbViClURecGQ=">ACAHicbZDNSsNAFIUn9a/Wv6gbwc1gEVyVRAR1IRTduKxgbaEN4WY6aedTMLMRCmhbnwVNy5U3PoY7nwbp20W2npg4OPce7lzT5BwprTjfFuFhcWl5ZXiamltfWNzy97euVNxKgmtk5jHshmAopwJWtdMc9pMJIUo4LQRDK7G9cY9lYrF4lYPE+pF0BUsZAS0sXx7D3yGL3BbpZHfxw9+xvojDAZ9u+xUnInwPLg5lFGum9/tTsxSMqNOGgVMt1Eu1lIDUjnI5K7VTRBMgAurRlUEBElZdNLhjhQ+N0cBhL84TGE/f3RAaRUsMoMJ0R6J6arY3N/2qtVIdnXsZEkmoqyHRmHKsYzyOA3eYpETzoQEgkpm/YtIDCUSb0EomBHf25HmoH1fOK+7NSbl6madRPvoAB0hF52iKrpGNVRHBD2iZ/SK3qwn68V6tz6mrQUrn9lFf2R9/gBTDZW6</latexit> <latexit sha1_base64="lLa6sjSOQSA6ZvKbViClURecGQ=">ACAHicbZDNSsNAFIUn9a/Wv6gbwc1gEVyVRAR1IRTduKxgbaEN4WY6aedTMLMRCmhbnwVNy5U3PoY7nwbp20W2npg4OPce7lzT5BwprTjfFuFhcWl5ZXiamltfWNzy97euVNxKgmtk5jHshmAopwJWtdMc9pMJIUo4LQRDK7G9cY9lYrF4lYPE+pF0BUsZAS0sXx7D3yGL3BbpZHfxw9+xvojDAZ9u+xUnInwPLg5lFGum9/tTsxSMqNOGgVMt1Eu1lIDUjnI5K7VTRBMgAurRlUEBElZdNLhjhQ+N0cBhL84TGE/f3RAaRUsMoMJ0R6J6arY3N/2qtVIdnXsZEkmoqyHRmHKsYzyOA3eYpETzoQEgkpm/YtIDCUSb0EomBHf25HmoH1fOK+7NSbl6madRPvoAB0hF52iKrpGNVRHBD2iZ/SK3qwn68V6tz6mrQUrn9lFf2R9/gBTDZW6</latexit> <latexit sha1_base64="lLa6sjSOQSA6ZvKbViClURecGQ=">ACAHicbZDNSsNAFIUn9a/Wv6gbwc1gEVyVRAR1IRTduKxgbaEN4WY6aedTMLMRCmhbnwVNy5U3PoY7nwbp20W2npg4OPce7lzT5BwprTjfFuFhcWl5ZXiamltfWNzy97euVNxKgmtk5jHshmAopwJWtdMc9pMJIUo4LQRDK7G9cY9lYrF4lYPE+pF0BUsZAS0sXx7D3yGL3BbpZHfxw9+xvojDAZ9u+xUnInwPLg5lFGum9/tTsxSMqNOGgVMt1Eu1lIDUjnI5K7VTRBMgAurRlUEBElZdNLhjhQ+N0cBhL84TGE/f3RAaRUsMoMJ0R6J6arY3N/2qtVIdnXsZEkmoqyHRmHKsYzyOA3eYpETzoQEgkpm/YtIDCUSb0EomBHf25HmoH1fOK+7NSbl6madRPvoAB0hF52iKrpGNVRHBD2iZ/SK3qwn68V6tz6mrQUrn9lFf2R9/gBTDZW6</latexit> Forward Pass Input pattern Output pattern Input 0 w 00 w 01 Input 1 Output 0 w 02 Input 2 Output 1 w 03 Input 3 X a i = w ij a j j
Forward Pass Input pattern Output pattern Input 0 w 00 w 01 Input 1 Output 0 w 02 Input 2 Output 1 inp 0 inp 1 w 03 inp 2 inp 3 Input 3 out 0 w 00 w 01 w 02 w 03 out 1 w 10 w 11 w 12 w 13 out = W inp
Matrix Representation w 00 w 01 w 02 w 03
How do we learn the weights? w 00 w 01 w 02 w 03
How do we learn the weights? w 00 Learning rules: w 01 1. Hebb’s rule w 02 2. Delta rule w 03
Hebb’s Rule • “Neurons that fire together wire together”
<latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> Hebb’s Rule • “Neurons that fire together wire together” target ( v ) 1 1 1 -1 1 -1 input ( u ) u ⊗ v 1 -1
<latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> <latexit sha1_base64="ghmEL9Yr4Y58wUGVf3yAcHa8M5U=">ACBHicbVBNS8NAEN34WetX1KMeFovgqSQiqLeiF48VjC20oWy2m3bpZhN2J8UScvHiX/HiQcWrP8Kb/8ZtG0FbHw83pthZl6QCK7Bcb6shcWl5ZXV0lp5fWNza9ve2b3Tcao82gsYtUMiGaCS+YB8GaiWIkCgRrBIOrsd8YMqV5LG9hlDA/Ij3JQ04JGKljH7SB3UMQZmnejoFHTOMfZh37IpTdSbA8QtSAUVqHfsz3Y3pmnEJFBtG65TgJ+RhRwKlhebqeaJYQOSI+1DJXE7POzyRc5PjJKF4exMiUBT9TfExmJtB5FgemMCPT1rDcW/NaKYTnfsZlkgKTdLoTAWGI8jwV2uGAUxMoRQxc2tmPaJIhRMcGUTgjv78jzxTqoXVfmtFK7LNIoX10iI6Ri85QDV2jOvIQRQ/oCb2gV+vRerberPdp64JVzOyhP7A+vgFYUJlR</latexit> Hebb’s Rule • “Neurons that fire together wire together” target ( v ) Storing a memory 1 1 1 -1 1 1 1 1 -1 -1 -1 -1 -1 1 input ( u ) u ⊗ v 1 1 1 1 -1 -1 -1 -1 -1 1
Recommend
More recommend