Autonomous Robots: Towards Founded Assessment of Robustness Gerald Steinbauer (1), Lisa-Christina Winter (2), Alexander Nussbaumer (2), and Dietrich Albert (2,3) (1) Institute for Software Technology, Graz University of Technology, Austria (2) Knowledge Technologies Institute, Graz University of Technology, Austria (3) Department of Psychology, University of Graz, Austria Presented by Alexander Nussbaumer SDIR Workshop @ ICRA Conference Karlsruhe, Germany 6 Mai 2013
Problem Statement • How can we guarantee that – a robot is dependable in the sense that it is able to successfully perform a task – in a non-deterministic dynamic environment – and facing internal and external difficulties Sensing Failures Perception Exogenous Incomplete or Decision Event Wrong Knowledge Environment Making Acting Hardware Failures Software Failures Execution Failures 2013-05-06 SDIR Workship @ ICRA Conference 2
Problem Statement Example: task to bring the letter to room B and the folder to C Exogenous Fault Execution Fault Sensing Fault 2013-05-06 SDIR Workship @ ICRA Conference 3
Problem Statement Challenges • develop algorithms that make a robot dependable when facing complex tasks, nondeterministic environments, and failures? – is done in the robotics group • evaluate, quantify and compare the dependability of such systems in the context of a very high number of varying environment instances, tasks, difficulties and faults ? – research task described in this paper 2013-05-06 SDIR Workship @ ICRA Conference 4
Problem Statement These days … • only a few formal approaches for evaluating performance and correctness for tasks available – NIST standard test methods for USAR robots (hardware centred) – workshop on conditions for replicable experiments and performance comparison in robotics research • no systematic, reliable, and valid assessment methods available for robots (?) • psychological sciences have well elaborated models and procedures to design and interpret capability tests for humans. – Item Response Theory (IRT) – Knowledge Space Theory (KST) 2013-05-06 SDIR Workship @ ICRA Conference 5
Objective and Idea long tradition of combine techniques, algorithms and psychological sciences in systems from the AI and assessing capabilities and robotics communities performances of individuals build build test framework for robots based on psychological methods for testing abilities and robustness of robots 2013-05-06 SDIR Workship @ ICRA Conference 6
Item Response Theory • most prominent example: PISA studies • probabilistic test theory for assessing abilities of people • test consists of items – question or problem to be answered; questionnaire • assessing latent abilities, e.g. certain mathematical skills – unidimensionality: only one ability is tested at the same time – e.g. in math word problems, only mathematical ability is tested (not reading ability) • assumption: stochastic relationship between the latent ability and the responses to items • different IRT models – e.g. Rasch Model: most common and empirically best evidenced • IRT has specific advantages – unidimensionality, specific objectivity, adaptive testing 2013-05-06 SDIR Workship @ ICRA Conference 7
Item Response Theory • constructing a test – definition of abilities to be tested – definition of items for the test – two types of parameter • person parameter ("ability") • item parameter ("difficulty") – method for parameter estimation • large sample of individuals required logistic function for one item – checking model fit • conducting a test – IRT test is easy to use and analyse – can be used in different contexts (schools, countries, etc.) – can be used adaptively 2013-05-06 SDIR Workship @ ICRA Conference 8
IRT for Robotics • problem := task + environment + failure – problems are used as IRT items – difficulty is defined by environmental setting and failures – tests needed if problems are solved or not • examples of problems – navigation to a target indoor on smooth surface – navigation to a target outdoor – navigation to a target outdoor with perception failure – navigation to a target in disaster area Environment – manipulation of rigid object – manipulation of glass with liquid • ability Task – abstract concept related to problems • robustness Failures – defined on abilities, difficulties, and mastered failures 2013-05-06 SDIR Workship @ ICRA Conference 9
IRT for Robotics • framework for testing robots on different levels Robustness Abilities Manipulation Object Navigation Recognition Performance Problems SDIR Workship @ ICRA Conference 2013-05-06 10
Robots and Environments • robot: PR2 – standard robot with clear set of abilities – common software architecture ROS – good simulation possibilities – different control architectures for performing tasks – variants within a control architecture (e.g. modification in perception) – other robots in future • environment: simulation – variable environments can easily be created in simulation – environment is controllable (difficulties, failures, …) 2013-05-06 SDIR Workship @ ICRA Conference 11
Procedure for Test Construction • definition abilities and problems – based on functions of robots – analysis of tasks based on experiences and literature review – different environments where tasks are performed • creation of individuals – control architectures and different variants • designing the empirical study – including failures • preparation of technical environment – simulation, tasks, individual robots, … • conducting empirical test • analysis – analysis of model fit – creation of assessment framework 2013-05-06 SDIR Workship @ ICRA Conference 12
Knowledge Space Theory • behavioristic mathematical-psychological theory – for representing a knowledge domain and the knowledge of a learner – problems are structured in prerequisite relations – directed acyclic graph – validation of structure in large samples – knowledge state: problems that can be soloved • allows for efficient assessment – adaptive assessment – not all problems have to be posed • KST in robotics – problems as defined before – e.g. if a robot can solve problem A that the robot can also solve problem B 2013-05-06 SDIR Workship @ ICRA Conference 13
Competence-based Knowledge Space T. • competence-based extension to KST – focus on underlying cognitive constructs in order to explain problem solving – assignment of competences to problems – structuring competences through prerequisite relations – validation of structure in large samples – competence state: competences that a person has • CbKST in robotics – abilities can be treated as competences – e.g. object recognition is a prerequisite for navigation 2013-05-06 SDIR Workship @ ICRA Conference 14
Conclusion • approach for creation of an assessment framework that allows fro testing abilities an robustness of robots – by combining methods from robotics and founded test theory from psychology • answering questions regarding – abilities of a specific robot – robustness of a specific robot – comparing robots • long-term vision is to create certification framework for robots 2013-05-06 SDIR Workship @ ICRA Conference 15
Next Steps • application for an Austrian national research project (FWF) • detailed elaboration of – conceptual and technical approach – ontology of problems, abilities, tasks, … • 3 phases: IRT - KST - CbKST – investigation how well they are applicable 2013-05-06 SDIR Workship @ ICRA Conference 16
Any Questions? Further Information: • Institute for Software Technology http://www.ist.tugraz.at/ • Cognitive Science Section @ KTI http://kti.tugraz.at/css/ 2013-05-06 SDIR Workship @ ICRA Conference 17
Open Discussion • Q1 : what are the problems of software development in robotics that can be considered "solved"? – none, in our opinion • Q2 : what are the solutions to these problems that can be considered best practice and why? – Robot systems are heavily related to integration of various hardware and software sub-systems. Modern frameworks and proved approaches ease the problem a little, for example the ROS hype. It eases the integration but has still a number of drawbacks • Q3 : to what extent the solutions to these problems are robotic-specific? – To a large extent integration is robotics-centered 2013-05-06 SDIR Workship @ ICRA Conference 18
Recommend
More recommend
