Sparse Attentive Backtracking: Temporal credit assignment through - PowerPoint PPT Presentation

Sparse Attentive Backtracking: Temporal credit assignment through reminding Nan Rosemary Ke 1,2 , Anirudh Goyal 1 , Olexa Bilaniuk 1 , Jonathan Binas 1 Chris Pal 2,4 , Mike Mozer 3 , Yoshua Bengio 1,5 1 Mila, Universit´ e de Montr´ eal 2 Mila, Polytechnique Montreal 3 University of Colorado, Boulder 4 Element AI 5 CIFAR Senior Fellow

Credit assignment • Credit assignment: The correct division and attribution of blame to one’s past actions in leading to a final outcome. • Credit assignment in recurrent neural networks uses backpropgation through time (BPTT). • Detailed memory of all past events • Assign soft credit to almost all past events • Diffusion of credit? 1

Credit assignment through time and memory • Humans selectively recall memories that are relevant to the current behavior. • Automatic reminding: • Triggered by contextual features. • Can serve a useful computation role in ongoing cognition. • Can be used for credit assignment to past events? • Assign credit through only a few states, instead of all states: • Sparse, local credit assignment. • How to pick the states to assign credit to? 2

Sparse Attentive Backtracking • Forward pass • Backward pass 3

Some results 4

Generalization and attention map • Generalization on longer sequences • Learned attention over different timesteps during training Copy Task with T = 200 5