reinforcement learning lecture 18a
play

Reinforcement Learning Lecture 18a Gillian Hayes 7th March 2007 - PowerPoint PPT Presentation

Aug 08, 2023 •388 likes •509 views

Reinforcement Learning Lecture 18a Gillian Hayes 7th March 2007 Gillian Hayes RL Lecture 18a 7th March 2007 1 Focussed Web Crawling Using RL Searching web for pages relevant to a specific subject No organised directory of web pages

  1. Reinforcement Learning Lecture 18a Gillian Hayes 7th March 2007 Gillian Hayes RL Lecture 18a 7th March 2007

  2. 1 Focussed Web Crawling Using RL • Searching web for pages relevant to a specific subject • No organised directory of web pages Web Crawling : start at one root page, follow links to other pages, follow their links to further pages, etc. Focussed Web Crawling : specific topic. Find maximum set of relevant pages having traversed minimum number of irrelevant pages. Why try this? : Less bandwidth, storage time (can take weeks for exhaustive search – billions of web pages) Good for dynamic content – can do frequent updates Can get indexing for a particular topic Alexandros Grigoriadis, MSc AI, Edinburgh 2003 + CROSSMARC project – extracting multilingual info from web on specific domains e.g. laptop retail info, job adverts on companies’ web pages Gillian Hayes RL Lecture 18a 7th March 2007

  3. 2 Web Crawler Retrieve Evaluate Good base set pages pages www Link Evaluate Extract queue links links RL link scorer • Link Queue: current set of links that have to be visited. Fetch link with highest score on queue Gillian Hayes RL Lecture 18a 7th March 2007

  4. 3 • Evaluate page this link points to: based on set of text/content attributes. If relevant, store on Good Pages • Get links from page • Evaluate links, add to link queue. Does does the link point to a relevant page? will it lead to relevant pages in future? • Where can we use RL? In the link scorer Gillian Hayes RL Lecture 18a 7th March 2007

  5. 4 RL Crawling • Reward when it finds relevant pages • Needs to recognise important attributes and follow most promising links first • Aim is to get π ∗ • How to formulate problem? What are states? What are actions? Alternatives: • State = a link, Action = { follow, don’t follow } • State = web page, Action = links • Learn V? Must do local search to get policy • Learn Q? More training examples needed since Q(s,a). But faster to use Choice: Action–links and learn V using TD( λ ) Gillian Hayes RL Lecture 18a 7th March 2007

  6. 5 How to Characterise a State? • Use text analyser to come up with keywords for domain – these words typically appear on web pages on this subject area • Feature vector of 500 binary attributes: existence or not of a keyword • State space: 2 500 states ∼ 10 150 – too large for a table • Use a neural network for function approximation to give V(s) • Learn weights of network using temporal difference learning • Eligibility trace on weights instead of states • Reward is 1/0 if page is/is not relevant Gillian Hayes RL Lecture 18a 7th March 2007

  7. 6 State Values V Tabular S V V(s) table Feature V(f) = f(s) S V(s) vector V(f(s)) encoding network Gillian Hayes RL Lecture 18a 7th March 2007

  8. 7 Learning Procedure • Use a number of training sets of web pages, e.g. different companies’ web sites containing numbers of pages with job adverts and start with a random policy • Learn V π , need to do GPI to get V ∗ • Then incorporate into a regular crawler: the RL neural net evaluates each page – the V value is its score • Which link to choose? Must do one-step lookahead – follow all links in current page, evaluate the pages they lead to • Place new pages on link queue according to score • Follow link at front of link queue to next page with highest likely relevance Gillian Hayes RL Lecture 18a 7th March 2007

  9. 8 Performance: Finds relevant pages (if > 1) following fewer links but searches more pages in the 1-step lookahead vs. CROSSMARC non-RL web crawler. Not so good at finding a single relevant page on a site. • Datasets: up to 2000 pages, 16000 links, tiny number of relevant pages in each dataset, English and Greek, 1000 training episodes Gillian Hayes RL Lecture 18a 7th March 2007

  10. 9 Issues Depends on: graphical structure of pages • Features chosen: many attributes were == 0 so not discriminating enough • Need to try on bigger datasets • Paper outlines alternative learning procedures Andrew McCallum’s CORA – searching computer science research papers • Treated roughly as a bandit problem learning Q(a). Action a = link on a web page and words in its neighbourhood • Choose the link expected to give highest future discounted reward • 53,000 documents, half a million links, 3x increase in efficiency (no. links followed before 75% of docs found vs. breadth-first search) Gillian Hayes RL Lecture 18a 7th March 2007

  11. 10 Alexandros Grigoriadis, Georgios Paliouras: Focused crawling using temporal difference-learning. Proceedings of the Panhellenic Conference in Artificial Intelligence (SETN), Lecture Notes in Artificial Intelligence 3025, 142–153, Springer-Verlag, 2004. Andrew McCallum et al.: Building domain-specific search engines with ML techniques. Proc AAAI-99 Spring Symposium on Intelligent Agents in Cyberspace Gillian Hayes RL Lecture 18a 7th March 2007

Recommend

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Refresh Your Understanding: Multi-armed Bandits Select all that are true: Up to slight variations

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

580 views • 55 slides
Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Refresh Your Understanding: Multi-armed Bandits Select all that are true: UCB selects the arm

Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With some slides derived from David Silver Lecture 12: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

758 views • 62 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Refresh Your Knowledge Fast RL Part II The prior over arm 1 is Beta(1,2) (left) and arm 2 is a

Refresh Your Knowledge Fast RL Part II The prior over arm 1 is Beta(1,2) (left) and arm 2 is a

Lecture 13: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With a few slides derived from David Silver Lecture 13: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement Learning ) Winter 2020

762 views • 40 slides
Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Refresh Your Knowledge. Policy Gradient Policy gradient algorithms change the policy parameters

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2020 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

719 views • 40 slides
Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement

Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement

Introduction CSCE CSCE 496/896 496/896 Lecture 7: Lecture 7: Reinforcement Reinforcement CSCE 496/896 Lecture 7: Learning Learning Consider learning to choose actions, e.g., Stephen Scott Reinforcement Learning Stephen Scott Robot

433 views • 9 slides
Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Table of Contents Behavioral Cloning 1 Inverse Reinforcement Learning 2 Apprenticeship

Lecture 7: Imitation Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With slides from Katerina Fragkiadaki and Pieter Abbeel Lecture 7: Imitation Learning 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 /

1.27k views • 45 slides
Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A.

Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A.

Reinforcement learning Applied artificial intelligence (EDA132) Lecture 13 2012-04-26 Elin A. Topp Material based on course book, chapter 21 (17), and on lecture Belningsbaserad inlrning / Reinforcement learning by rjan Ekeberg,

1.39k views • 126 slides
Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

CSE/NB 528 Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13 1 Todays Agenda: All about Learning F Unsupervised Learning Sparse Coding Predictive Coding F Supervised learning Perceptrons and

479 views • 24 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to

Lecture 1: Introduction to Reinforcement Learning Introduction to Reinforcement Learning Kevin Chen and Zack Khan Lecture 1: Introduction to Reinforcement Learning Outline 1. Course Logistics 2. What is Reinforcement Learning? 3.

930 views • 67 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and

Lecture 14: Introduction to Reinforcement Learning CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline What is Reinforcement Learning RL Formalism 1. Reward 2. The agent 3. The environment 4. Actions 5.

1.27k views • 64 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q

CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q

CSCE 496/896 Lecture 7: Reinforcement CSCE 496/896 Lecture 7: Learning Stephen Scott Reinforcement Learning Introduction MDPs Q Learning Stephen Scott TD Learning DQN (Adapted from Paul Quint) Atari Example Go Example

637 views • 53 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last Lecture v What is reinforcement learning? v Difference from other AI problems v Application

1.04k views • 52 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall

This lecture will be recorded!!! Welcome to DS595/CS525 Reinforcement Learning Prof. Yanhua Li Time: 6:00pm 8:50pm R Zoom Lecture Fall 2020 Last lecture v Reinforcement Learning Components Model, Value function, Policy v Model-based

944 views • 78 slides

More recommend