learning and sharing knowledge for robots
play

Learning(and(Sharing(Knowledge(( for(Robots learn Interact - PowerPoint PPT Presentation

Jul 22, 2023 •157 likes •617 views

Learning(and(Sharing(Knowledge(( for(Robots learn Interact Ashutosh'Saxena' ' share CEO,'Brain'of'Things' ' ' Prepare&affogato: ' !Take!some!coffee!in!a!cup.!Add!ice!cream!of!your!choice.!!!

  1. Learning(and(Sharing(Knowledge(( for(Robots learn Interact Ashutosh'Saxena' ' share CEO,'Brain'of'Things' ' '

  2. Prepare&affogato: '“ !Take!some!coffee!in!a!cup.!Add!ice!cream!of!your!choice.!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Finally!add!raspberry!syrup!to!the!mixture. ”' 2' Sung,'Jin,'Saxena:'Robobarista.''Misra'et'al.,'Tell'Me'Dave' Ashutosh'Saxena'

  3. Scene&Understanding& • fridge& [Koppula'&'Saxena'et'al.'2011]' microwave& [Socher'et'al.'2011]' oven& [Yao'et'al.'2012]' [Farabet'et'al.'2013]' stove& [Wu'et'al.'2014]' 0%' 20%'40%'60%'80%' 100%' …' ' Natural&Language&Understanding ' Moveto ($x) Grasp ($x) Toggle ($x, on) … [Walter'et'al.'2013]' [Misra,'Sung,'Saxena'2014]' [Beetz'et'al.'2011]' Manipula>on&Planning& Robot& ! LearningXbased'Approaches' ! MoWon'Planners' '' PRM*'/'RRT*' [Karaman'&'Frazzoli'2011]'' '' Markov'Decision'Process'(MDP)' ''' CHOMP' [Ratliff'et'al.'2009]' ''Inverse'reinforcement'learning'(ILR)'' ''''''' [Ng'&'Russel'2000;'Abbeel'&'Ng'2004] ' ''' trajopt' [Schulman'et'al.'2013] ]' ''Max'Margin'Planning' [Ratliff'et'al.'2006]' [Muelling'et'al.'2010]' [MaiWnXShepard'et'al.'2010]' Sung,'Jin,'Saxena:'Robobarista,'ISRR'2015' 3'

  4. Deep'Learning' Robot& 4'

  5. Robot& Structured' Deep'Learning' 5'

  6. Recent(progress(in(deep(learning Krizhevsky'et'al.'NIPS’12' “Cat”' X' &'many'more' Y' ruth Image FCN-8s Long'et'al.'CVPR’15' &'many'more' “Hallo'welt”' Sutskever'et'al.'NIPS’14' “Hello'world”' &'many'more' Vinyals'et'al.'CVPR’15' A group of people shopping at an &'many'more' outdoor market. ! “Hello'world”' Hinton'et'al.'IEEE'speech’12' &'many'more' 6' Ashutosh'Saxena'

  7. Structures(of(deep(architectures Hierarchical&RNN& Du'et'al.'CVPR’15' ' Recursive&Neural&Networks& Socher'et'al.'ICML’11' Structured&predic>on&with&deep&neural&networks& Chen'et'al.'ICLR’15' Zhang'et'al.'CVPR’14' Tompson'et'al.'SIGGRAPH’14' Chen'et'al.'ICML’15' Zheng'et'al.'ICCV’15' ' Combining!structure!with!deep!neural!networks!helps! 7' Ashutosh'Saxena'

  8. Interact,(Learn,(and(Share Learn!shared! Robots!interact!with! models!and! the!world,!humans! representaBons! and!Internet! learn ! Interact share Update!the! knowledge!in!the! cloud!(RoboBrain)! 8' Ashutosh'Saxena'

  9. Robot(Tasks(and(Interac=on Robot&Language& Manipula>ng&food& Appliance&Manipula>on& Tell&Me&Dave,& RSS’13.' DeepMPC.'Lenz,'Knepper,' RoboBarista,& ' Tellex'et'al.' Saxena,'RSS'2014' Sung'et'al.,'2015' Homes&and&Humans& Human&object&interac>on& Brain'of'Things,'2015' Koppula'et'al.'T.'PAMI’15' Abbeel'et'al.,'UC'Berkeley' Li'et'al.'ECCV’08' Gupta'et'al.'T.'PAMI’09' 9' Ashutosh'Saxena'

  10. Why(is(it(challenging? ! Large'variety'in' environments' ! Large'variety'of' objects' ! Large'variety'of' required'movements' Sung,'Jin,'Saxena:'Robobarista' 10' Ashutosh'Saxena'

  11. Manipula=on(Mo=on(Planning • Many'objects'share'similar'parts'operated'in' similar'manner' Even'if'the'robot'has'not'seen'the'object'before,' prior'moWon'plan'can'be'reXused'on'new'objects.' Sung,'Jin,'Saxena:'Robobarista' 11' Ashutosh'Saxena'

  12. Manipula=on(Mo=on(Planning • Espresso'machine' Sung,'Jin,'Saxena:'Robobarista' 12' 6 Ashutosh'Saxena'

  13. Instruc=on(Manual(Instan=a=on • Transfer'from'similar'object'parts' Sung,'Jin,'Saxena:'Robobarista' 13' Ashutosh'Saxena'

  14. Robot(Demo:(making(laGe Sung,'Jin,'Saxena:'Robobarista,'ISRR'2015' 14'

  15. Dataset Accuracy&(%)&& [DTWNMT&<&10]& chance! ! 116'objects'(250'parts)' object!part!classifier! • KinectFusion' LSSVM!+!kinemaBc!structure![50]! ! 1225'crowdXsourced'trajectories' similar!task!+!weighBng![13]! Our'Model'without'NoiseXhandling' • 71'nonXexpert'Mechanical'Turkers' Our$Model$ • Robobarista'Plarorm'' Sung,'Jin,'Saxena:'Robobarista,'ISRR'2015' 15'

  16. Mul=modal(data:(Language,( Trajectories,(PointIcloud Ashutosh'Saxena' Ashutosh'Saxena'

  17. Robots that Learn Ashutosh'Saxena' Ashutosh'Saxena'

  18. RoboBrain snapshot RoboBrain'Knowledge'learned'from:' • Watching'(YouTube'videos,'WikiHow,'Wikipedia,…)' • InteracWng'(online'games,'physical'robots)' Ashutosh'Saxena' Ashutosh'Saxena'

  19. Percep=on Ashutosh'Saxena'

  20. Real(world(driving(examples 1100'miles'of'driving'data'from'10'drivers' • Cameras'(inside'&'outside),''GPS,'Vehicle'dynamics' • Brain4Cars,'Jain'et'al.' 20' Ashutosh'Saxena'

  21. FusionRNN(for(an=cipa=on Training' example' Learning'' Learning'' “Memory”' to'fuse' to'anWcipate' Fig. 4: Sensory fusion RNN for anticipation. ( Bottom ) In the Jain'et'al.'ICRA’16' Brain4Cars,'Jain'et'al.' 21' Ashutosh'Saxena'

  22. Results TimeNtoN Precision& Recall& maneuver&(s)& Chance' 20.0' 20.0' XX' SVM' 43.7' 37.7' 1.20' Morris'et'al.'IVS’11' SimpleRNN' 78.0' 71.1' 3.15' FusionRNN' 84.5& 77.1& 3.58' Jain'et'al.'ICRA’16' 22' Jain'et'al.'ICCV’15,'ICRA’16,'IJRR’16*' Ashutosh'Saxena'

  23. Forces,(Contact(Manipula=on DeepMPC,'Lenz,'Knepper'and'Saxena,'RSS'2014'

  24. Robot(Tasks(and(Interac=on Robot&Language& Manipula>ng&food& Appliance&Manipula>on& Tell&Me&Dave,& RSS’13.' DeepMPC.'Lenz,'Knepper,' RoboBarista,& ' Tellex'et'al.' Saxena,'RSS'2014' Sung'et'al.,'2015' Homes&and&Humans& Human&object&interac>on& Brain'of'Things,'2015' Koppula'et'al.'T.'PAMI’15' Abbeel'et'al.,'UC'Berkeley' Li'et'al.'ECCV’08' Gupta'et'al.'T.'PAMI’09' 24' Ashutosh'Saxena'

  25. Interact,(Learn,(and(Share Learn!shared! Robots!interact!with! models!and! the!world,!humans! representaBons! and!Internet! learn ! Interact share Update!the! knowledge!in!the! cloud!(RoboBrain)! 25' Ashutosh'Saxena'

  26. Spa=oItemporal(problems Human'object'interacWon'' Modeling'human'moWon' (Koppula'et'al.'IJRR’13)' '(Taylor'et'al.'NIPS’06)' Douillard'et'al.'ISRR’07' Input 𝑌 � 𝑌 � 𝑌 � 𝑌 � Li'et'al.'ECCV’08' Layer Outside features Lezama'et'al.'CVPR’11' Hidden ? 𝑍 𝑍 𝑍 𝑍 � � � � Driver states Layer Fragkiadaki'et'al.'ICCV’15' Inside features Koppula'et'al.'RSS’13' Output 𝑎 � 𝑎 � 𝑎 � 𝑎 � Layer 𝑈 Jain'et'al.'ICCV’15' Maneuver'anWcipaWon' Tracking' Brendel'et'al.'ICCV’11' (Jain'et'al.'ICCV’15)' (Li'et'al.'ECCV’08)' …'and'many'more' 26' Ashutosh'Saxena'

  27. StructuralIRNN *Scalable'and'flexible' ' *Generic'and'principled' jhjh' ' *EndXtoXend'trainable' ' StructuralXRNN:'Deep'Learning'on'SpaWoXTemporal'Graphs.'Jain,'Zamir,' 27' Savarese,'Saxena.'In'CVPR,'2016.' Ashutosh'Saxena'

  28. High(level(steps(for(transforming 1.'FactorXgraph'parameterizaWon' ' ' 2.'SemanWcally'parWWoning'the'nodes' ' ' 3.'Modeling'each'factor'funcWon'with'RNN' ' ' 4.'Wiring'the'RNNs'to'form' structuralNRNN& 28' Ashutosh'Saxena'

  29. 1.(FactorIgraph(parameteriza=on • FactorXgraph'defines'a'funcWon'over'spaWoXtemporal'graph' • Factor!funcBon!captures!interacBons!between!nodes! ' Factor graph 29' Ashutosh'Saxena'

  30. 2.(FactorIsharing • SemanWcally'similar'nodes/edges'may'share'the' • FactorXgraph'defines'a'funcWon'over'spaWoXtemporal'graph' • Factor!funcBon!captures!interacBons!between!nodes! factor'funcWon'and'parameters.' ' Factor graph Why'share'factors?'' Incorporate'priors,'compactness,'flexibility,'generalizaWon' 30' Ashutosh'Saxena'

  31. 2.(FactorIsharing GeneralizaBon!to!changes!in!environment!and!task! 31' Ashutosh'Saxena'

  32. 3.(Modeling(factors(with(RNNs • Flexibility'in'designing'each'RNN' • nodeRNNs'and'edgeRNNs'are'connected'to'capture'the' spaWoXtemporal'interacWons' 32' Ashutosh'Saxena'

  33. 4.(StructuralIRNN Algorithm 1 From spatio-temporal graph to S-RNN Input G = ( V , E S , E T ) , C V = { V 1 , ..., V P } Output S-RNN graph G R = ( { R E m } , { R V p } , E R ) 1: Semantically partition edges C E = { E 1 , ..., E M } 2: Find factor components { Ψ V p , Ψ E m } of G 3: Represent each Ψ V p with a nodeRNN R V p 4: Represent each Ψ E m with an edgeRNN R E m 5: Connect { R E m } and { R V p } to form a bipartite graph. ( R E m , R V p ) ∈ E R iff ∃ v ∈ V p , u ∈ V s.t. ( u, v ) ∈ E m Return G R = ( { R E m } , { R V p } , E R ) StructuralXRNN'is'a' biparWte'graph' 33' Ashutosh'Saxena'

  34. Training(StructuralIRNN Parameter&sharing 'through' ' Structured&feature&space:& Input'in''''''''''is'' ' ' ForwardXpass'for' v! ForwardXpass'for' u! ForwardXpass'for'object'node' w! ForwardXpass'for'human'node' v! Ashesh'Jain' 34'

  35. Key(takeaways • SpaWoXtemporal'interacWons'are'captured'by'the' connecWons'between'nodeRNNs'and'edgeRNNs' • Sharing'edgeRNNs'learns'dependencies'between' the'output'labels' • Structured'feature'space'' 35' Ashutosh'Saxena'

Recommend

Discovering, Visualizing and Sharing Knowledge through Personalized Learning Knowledge Maps The

Discovering, Visualizing and Sharing Knowledge through Personalized Learning Knowledge Maps The

Discovering, Visualizing and Sharing Knowledge through Personalized Learning Knowledge Maps The AWAKE Project personal knowledge structure (user A) A m E ?? y map grid semantische ? C d achsen o netz awarenes cs kuns s multi-

234 views • 21 slides
ESCRI-SA Knowledge Sharing Sharing Objectives and Components A presentation for the ESCRI-SA

ESCRI-SA Knowledge Sharing Sharing Objectives and Components A presentation for the ESCRI-SA

ESCRI-SA Knowledge Sharing Sharing Objectives and Components A presentation for the ESCRI-SA Knowledge Sharing Reference Group, Meeting 1 February 6, 2018 Knowledge Sharing Plan (KSP) The sharing of knowledge from ARENA funded projects

197 views • 6 slides
Building Knowledge &amp; Networking Capacities How to Succeed and Fail at Knowledge Sharing

Building Knowledge &amp; Networking Capacities How to Succeed and Fail at Knowledge Sharing

Building Knowledge &amp; Networking Capacities How to Succeed and Fail at Knowledge Sharing Cheryl Cooper, Applied Wisdom International Knowledge &amp; Learning Consultant CherylAppliedWisdom@gmail.com Since 2002... Strategic Knowledge

490 views • 23 slides
Accelerating Innovation through Knowledge &amp; Learning Transferring a Knowledge Management

Accelerating Innovation through Knowledge &amp; Learning Transferring a Knowledge Management

Accelerating Innovation through Knowledge &amp; Learning Transferring a Knowledge Management proof of concept model from the for-Profit sector to the not-for-Profit sector and driving multi- stakeholder cross- sector knowledge sharing

253 views • 23 slides
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine

Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine

Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation Poorya ZareMoodi , Wray Buntine, Gholamreza (Reza) Haffari Monash University Slides : Roadmap ! 2 Introduction &amp; background Adaptive

360 views • 17 slides
Knowledge Sharing If more than one person is building a knowledge base, they must be able to

Knowledge Sharing If more than one person is building a knowledge base, they must be able to

Knowledge Sharing If more than one person is building a knowledge base, they must be able to share the conceptualisation. A conceptualization is a map from the problem domain into the representation. A conceptualization specifies: What

426 views • 6 slides
Learning, Equilibria, Limitations, and Robots * Michael Bowling Computer Science Department

Learning, Equilibria, Limitations, and Robots * Michael Bowling Computer Science Department

Learning, Equilibria, Limitations, and Robots * Michael Bowling Computer Science Department Carnegie Mellon University *Joint work with Manuela Veloso Talk Outline Robots A two robot, adversarial, concurrent learning problem. The

467 views • 46 slides
Tools and Strategies for Sharing Knowledge Peggy D Adamo The INFO Project Johns Hopkins

Tools and Strategies for Sharing Knowledge Peggy D Adamo The INFO Project Johns Hopkins

Tools and Strategies for Sharing Knowledge Peggy D Adamo The INFO Project Johns Hopkins Bloomberg School of Public Health Center for Communication Programs 1 Knowledge Sharing Tools &amp; Strategies Why should we share knowledge? 2

409 views • 28 slides
and sharing agroecological knowledge Jorge Chavez-Tafur, Paulo Petersen, Frank van Schoubroeck

and sharing agroecological knowledge Jorge Chavez-Tafur, Paulo Petersen, Frank van Schoubroeck

Documentation for building and sharing agroecological knowledge Jorge Chavez-Tafur, Paulo Petersen, Frank van Schoubroeck and Edith van Walsum IAALD XIIIth World Congress, Montpellier, 26-29 April 2010 ileia Centre for Learning on

411 views • 28 slides
Academia: no learning without knowledge, no knowledge without learning TRPC6, NEP Van den Berg,

Academia: no learning without knowledge, no knowledge without learning TRPC6, NEP Van den Berg,

Academia: no learning without knowledge, no knowledge without learning TRPC6, NEP Van den Berg, Weening et al., 2004 Pathology of Idiopathic Membranous Glomerulopathy Subepithelial electron dense deposits Injury of epithelial cells

303 views • 11 slides
Temporal Analysis of Inter-Community User Flows in Online Knowledge-Sharing Networks Anna

Temporal Analysis of Inter-Community User Flows in Online Knowledge-Sharing Networks Anna

Temporal Analysis of Inter-Community User Flows in Online Knowledge-Sharing Networks Anna Guimar aes, Ana Paula Couto da Silva, Jussara Almeida Department of Computer Science - UFMG (Brazil) August 10, 2015 Communities in Knowledge-Sharing

513 views • 32 slides
ESCRI-SA KNOWLEDGE SHARING Second Workshop 8 May 2018 HIGH VOLTAGE SOLUTIONS 1 Agenda

ESCRI-SA KNOWLEDGE SHARING Second Workshop 8 May 2018 HIGH VOLTAGE SOLUTIONS 1 Agenda

ESCRI-SA KNOWLEDGE SHARING Second Workshop 8 May 2018 HIGH VOLTAGE SOLUTIONS 1 Agenda Welcome Introduce CPP and ABB Construction Video Knowledge Sharing CPP and ABB CPP: Key Achievements CPP: Installation Testing

255 views • 24 slides
Personal Knowledge Elaboration and Sharing: Presentation is Knowledge too Article January 2008

Personal Knowledge Elaboration and Sharing: Presentation is Knowledge too Article January 2008

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/239832574 Personal Knowledge Elaboration and Sharing: Presentation is Knowledge too Article January 2008 CITATIONS READS 2 53 3

187 views • 18 slides
Ethics and Authority Sharing for Autonomous Armed Robots RDA2 ECAI 2012 workshop on Right and

Ethics and Authority Sharing for Autonomous Armed Robots RDA2 ECAI 2012 workshop on Right and

Ethics and Authority Sharing for Autonomous Armed Robots RDA2 ECAI 2012 workshop on Right and Duties of Autonomous Agents Florian Gros, ONERA Catherine Tessier, ONERA Thierry Pichevin, Centre de recherche des Ecoles de Saint-Cyr Cotquidan

380 views • 15 slides
M edical knowledge sharing and information exchange: virtually enabling access to specialty care

M edical knowledge sharing and information exchange: virtually enabling access to specialty care

M edical knowledge sharing and information exchange: virtually enabling access to specialty care Dr Dana Newcomb Medical Director Integrated Care Childrens Health Queensland Project ECHO Superhub M edical knowledge sharing and information

330 views • 9 slides
Learning Robots Pavel Petrovi Department of Applied Informatics, Faculty of Mathematics,

Learning Robots Pavel Petrovi Department of Applied Informatics, Faculty of Mathematics,

Learning Robots Pavel Petrovi Department of Applied Informatics, Faculty of Mathematics, Physics and Informatics ppetrovic@acm.org August 2009 Life is learning... :-) LEGO Geometry Learning Robots, August 2009 2 But how does the learning

1.31k views • 109 slides
Health Sciences The challenges of sharing knowledge across boundaries: tales from an NIHR

Health Sciences The challenges of sharing knowledge across boundaries: tales from an NIHR

Leeds Institute of Health Sciences The challenges of sharing knowledge across boundaries: tales from an NIHR Knowledge Mobilisation Research Fellow Dr Vicky Ward KM: a brief (and imperfect) history 1990s Evidence based practice and policy

520 views • 12 slides
Knowledge Transfer Between Robots with Similar Dynamics for High-Accuracy Impromptu Trajectory

Knowledge Transfer Between Robots with Similar Dynamics for High-Accuracy Impromptu Trajectory

Knowledge Transfer Between Robots with Similar Dynamics for High-Accuracy Impromptu Trajectory Tracking European Control Conference June 26, 2019 SiQi Zhou 1 , Andriy Sarabakha 2 , Erdal Kayacan 3 , Mohamed K. Helwa 1 , and Angela P. Schoellig 1

387 views • 25 slides
2014 BRAZILS NATIONAL SANITATION QUALITY AWARDS (PNQS) Promoting Knowledge Sharing and

2014 BRAZILS NATIONAL SANITATION QUALITY AWARDS (PNQS) Promoting Knowledge Sharing and

2014 BRAZILS NATIONAL SANITATION QUALITY AWARDS (PNQS) Promoting Knowledge Sharing and Learning in Brazilian Water and Sanitation SYNOPSIS In the developing world, Brazil has become Quality Committee (CNQA). Tie PNQS Hub in partnership

452 views • 8 slides
Active Learning Passive Learning Active Learning 1. Think 1. Acquisition of knowledge Ability

Active Learning Passive Learning Active Learning 1. Think 1. Acquisition of knowledge Ability

Active Learning Passive Learning Active Learning 1. Think 1. Acquisition of knowledge Ability 2. Decision making 2. Application of knowledge 3. Explain/communicate 1. Deep learning Approach to Uni-directional 2. Interactive Learning

282 views • 5 slides
Name Name Learning for Doncaster Learning Learning for Sport England Sharing locally,

Name Name Learning for Doncaster Learning Learning for Sport England Sharing locally,

Name Name Learning for Doncaster Learning Learning for Sport England Sharing locally, regionally &amp; nationally Population PA is the norm Active places accessible for all Community shapes what it wants changes

316 views • 7 slides
Knowledge warehousing with WebKB-2 A knowledge server for structuring, sharing and publishing

Knowledge warehousing with WebKB-2 A knowledge server for structuring, sharing and publishing

1 Knowledge warehousing with WebKB-2 A knowledge server for structuring, sharing and publishing information Dr Philippe MARTIN philippe.martin@gu.edu.au The on-line slides are accessible at http://www.webkb.org/doc/slides/dstc0302.html 2 Plan

268 views • 15 slides
Approaches to Probabilistic Model Learning for Mobile Manipulation Robots Jrgen Sturm

Approaches to Probabilistic Model Learning for Mobile Manipulation Robots Jrgen Sturm

Approaches to Probabilistic Model Learning for Mobile Manipulation Robots Jrgen Sturm University of Freiburg (now at Technical University of Munich) PhD Supervisor: Wolfram Burgard Motivation What could flexible service robots do for us?

1.04k views • 79 slides
The OGP Civil Society Knowledge Sharing Workshop: Working

The OGP Civil Society Knowledge Sharing Workshop: Working

The OGP Civil Society Knowledge Sharing Workshop: Working with Government on NAPs Patrice McDermoC ExecuGve Director Jesse Franzblau Policy Associate

850 views • 12 slides

More recommend