Interlude 1 OpenAI GPT2 Language models unigrams, bigrams, - PDF document

Interlude 1 OpenAI – GPT2 § Language models – unigrams, bigrams, Markov models, ELMO § GPT2 – transformer based NN § 1.5 Billion parameters § Trained on 40 GB Internet text (no supervision) 2 1

Generate Synthetic Text Human Prompt: In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English . GPT continues… (best of 10 tries) The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four- horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. … 3 Generate Synthetic Text Human Prompt: A train carriage containing controlled nuclear materials was stolen in Cincinnati today. Its whereabouts are unknown GPT continues… (best of 10 tries) In an email to Ohio news outlets, the U.S. Department of Energy said it is working with the Federal Railroad Administration to find the thief. “The theft of this nuclear material will have significant negative consequences on public and environmental health, our workforce and the economy of our nation,” said Tom Hicks, the U.S. Energy Secretary, in a statement. “Our top priority is to secure the theft and ensure it doesn’t happen again.” The stolen material was taken from the University of Cincinnati’s Research Triangle Park nuclear research site, according to a news release from Department officials. The Nuclear Regulatory Commission did not immediately release any information. 4 2

Zero Shot Learning on other Tasks Winigrad Schema Challenge The trophy would not fit in the brown suitcase because it was too big ( small ). What was too big ( small )? Answer 0: the trophy Answer 1: the suitcase The town councilors refused to give the demonstrators a permit because they feared ( advocated ) violence. Who feared ( advocated ) violence? Answer 0: the town councilors Answer 1: the demonstrators GPT 70.7% accuracy Previous record: 63.7% Human: 92%+ 5 CSE P 573: Artificial Intelligence Reinforcement Learning Dan Weld/ University of Washington [Many slides taken from Dan Klein and Pieter Abbeel / CS188 Intro to AI at UC Berkeley – materials available at http://ai.berkeley.edu.] 3

Reinforcement Learning Reinforcement Learning Agent State: s Actions: a Reward: r Environment § Basic idea: § Receive feedback in the form of rewards § Agent’s utility is defined by the reward function § Must (learn to) act so as to maximize expected rewards § All learning is based on observed samples of outcomes! 4

Example: Animal Learning § RL studied experimentally for more than 60 years in psychology § Rewards: food, pain, hunger, drugs, etc. § Mechanisms and sophistication debated § Example: foraging § Bees learn near-optimal foraging plan in field of artificial flowers with controlled nectar supplies § Bees have a direct neural connection from nectar intake measurement to motor planning area Example: Backgammon § Reward only for win / loss in terminal states, zero otherwise § TD-Gammon learns a function approximation to V(s) using a neural network § Combined with depth 3 search, one of the top 3 players in the world § You could imagine training Pacman this way… § … but it ’ s tricky! (It ’ s also PS 4) 5

Example: Learning to Walk Initial [Kohl and Stone, ICRA 2004] [Video: AIBO WALK – initial] Example: Learning to Walk Finished [Kohl and Stone, ICRA 2004] [Video: AIBO WALK – finished] 6

Example: Sidewinding [Andrew Ng] [Video: SNAKE – climbStep+sidewinding] “Few driving tasks are as intimidating as parallel parking…. https://www.youtube.com/watch?v=pB_iFY2jIdI 15 7

Parallel Parking “Few driving tasks are as intimidating as parallel parking…. https://www.youtube.com/watch?v=pB_iFY2jIdI 16 Other Applications § Robotic control § helicopter maneuvering, autonomous vehicles § Mars rover - path planning, oversubscription planning § elevator planning § Game playing - backgammon, tetris, checkers, chess, go § Computational Finance, Sequential Auctions § Assisting elderly in simple tasks § Spoken dialog management § Communication Networks – switching, routing, flow control § War planning, evacuation planning, forest-fire treatment planning 8

Reinforcement Learning § Still assume a Markov decision process (MDP): § A set of states s Î S § A set of actions (per state) A § A model T(s,a,s’) § A reward function R(s,a,s’) & discount γ ? § Still looking for a policy p (s) § New twist: don’t know T or R § I.e. we don’t know which states are good or what the actions do § Must actually try actions and states out to learn Offline (MDPs) vs. Online (RL) Many people call this RL as well Simulator Offline Solution Monte Carlo Online Learning (Planning) Planning (RL) Diff: 1) dying ok; 2) (re)set button 9

Demo § Stanford Helicopter https://www.youtube.com/watch?v=Idn10JBsA3Q 20 Four Key Ideas for RL § Credit-Assignment Problem § What was the real cause of reward? § Exploration-exploitation tradeoff § Model-based vs model-free learning § What function is being learned? § Approximating the Value Function § Smaller à easier to learn & better generalization 10

Credit Assignment Problem 22 Exploration-Exploitation tradeoff § You have visited part of the state space and found a reward of 100 § is this the best you can hope for??? § Exploitation : should I stick with what I know and find a good policy w.r.t. this knowledge? § at risk of missing out on a better reward somewhere § Exploration : should I look for states w/ more reward? § at risk of wasting time & getting some negative reward 23 11

Model-Based Learning Model-Based Learning § Model-Based Idea: § Learn an approximate model based on experiences § Solve for values as if the learned model were correct § Step 1: Learn empirical MDP model § Explore (e.g., move randomly) § Count outcomes s’ for each s, a and estimate § Normalize to give an estimate of § Discover each when we experience (s, a, s’) § Step 2: Solve the learned MDP § For example, use value iteration, as before 12

Example: Model-Based Learning Random p Observed Episodes (Training) Learned Model Episode 1 Episode 2 T(s,a,s’). B, east, C, -1 B, east, C, -1 T(B, east, C) = 1.00 A T(C, east, D) = 0.75 C, east, D, -1 C, east, D, -1 T(C, east, A) = 0.25 D, exit, x, +10 D, exit, x, +10 … B C D R(s,a,s’). Episode 3 Episode 4 E R(B, east, C) = -1 E, north, C, -1 E, north, C, -1 R(C, east, D) = -1 C, east, D, -1 C, east, A, -1 R(D, exit, x) = +10 Assume: g = 1 D, exit, x, +10 A, exit, x, -10 … Convergence § If policy explores “enough” – doesn’t starve any state § Then T & R converge § So, VI, PI, Lao* etc. will find optimal policy § Using Bellman Equations § When can agent start exploiting?? § (We’ll answer this question later) 27 13

Two main reinforcement learning approaches § Model-based approaches: § explore environment & learn model, T=P( s ’ | s , a ) and R( s , a ), (almost) everywhere § use model to plan policy, MDP-style § approach leads to strongest theoretical results § often works well when state-space is manageable § Model-free approach: § don ’ t learn a model of T&R; instead, learn Q-function (or policy) directly § weaker theoretical results § often works better when state space is large 28 Two main reinforcement learning approaches § Model-based approaches: Learn T + R |S| 2 |A| + |S||A| parameters (40,400) § Model-free approach: Suppose 100 states, 4 actions Learn Q |S||A| parameters (400) 29 14

Model-Free Learning Nothing is Free in Life! § What exactly is Free??? § No model of T § No model of R § (Instead, just model Q) 31 15

Reminder: Q-Value Iteration § Forall s, a § Initialize Q 0 (s, a) = 0 no time steps left means an expected reward of zero § K = 0 § Repeat Q k+1 (s,a) do Bellman backups For every (s,a) pair: a s, a s,a,s’ V k (s’)=Max a’ Q k (s’,a’) K += 1 § Until convergence I.e., Q values don’t change much We know this…. We can sample this Puzzle: Q-Learning § Forall s, a § Initialize Q 0 (s, a) = 0 no time steps left means an expected reward of zero § K = 0 § Repeat Q k+1 (s,a) do Bellman backups For every (s,a) pair: a s, a s,a,s’ V k (s’)=Max a’ Q k (s’,a’) K += 1 § Until convergence I.e., Q values don’t change much Q: How can we compute without R, T ?!? A: Compute averages using sampled outcomes 16