How to Address the Approval Trap for Autonomous Vehicles A survey - PowerPoint PPT Presentation

Prof. Dr. rer. nat. Hermann Winner Maren Graupner, M.Sc. Dipl.-Ing. Walther Wachenfeld How to Address the Approval Trap for Autonomous Vehicles A survey of the challenge on safety validation and releasing the autonomous vehicle How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

2 The Approval Trap Assumption:  The development of autonomously driving vehicles has been finished. Safety requirement:  The risk with autonomous driving should not exceed the risk of conventional driving. Research Development Production Usage For release this vehicle has to get an approval (by authorities and/or company internal sign off).  A validation that the safety level complies to the requirement above has to be done before the release.  How can we validate that the risk does not exceed the current risk level? How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

3 Statistical Data for Risk Reference Num umbe bers for or Ger ermany 2013 2013 Total Autobahn Distance travelled/10 9 km 724 225 2.42 0.15 Total number of accidents/10 6 With personal injury/10 3 291 18.4 Distance between two accidents/10 6 km: All accidents 0.34 1.67 Involving personal injury 2.5 12 Involving serious casualties >11 >40 Involving fatalities >200 660 Reference distances as a function of the area of use and the consequences of accidents (Source: [Statistisches Bundesamt, German Federal Statistical Office, 2014]) Note: - Regarding information on seriously injured casualties and fatalities, the figures refer to distance per person. However, as more than one person per accident is affected in the category, the given value is a lower estimate. - Only non-urban highways How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

4 Risk Figures for Human Drivers 1 Person drives 55 years independently, each year 14,000 km = 770,000 km/lifetime ≈ 15,000 h (at average speed 50 km/h)  The average driver is involved every 340,000 km into a reported accident, and self caused by 60%.  The average driver is involved every 210 mio. km into an accident with fatalities.  In average 1.4 accidents are caused by one human in his/her lifetime and 1:450 fatalities/lifetime. Although human drivers make millions of mistakes very, very few severe accidents occur as a consequence. How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

5 Known Approval Strategies Endurance driving test  Proof of relative risk to drive a specific distance in which the expected event (crash) is statistically occurring multiple times.  State-of-the-Art for ADAS: 1 … 10 millions km  Driver is still responsible and ADAS is designed Source: KFZTicker.de for controlability What are the necessary travel distances for an approval of autonomous driving? (Source: [Fach, M., Baumann, F., Breuer, J., May, A.: Bewertung der Beherrschbarkeit von Aktiven Sicherheits- und Fahrerassistenzsystemen an den Funktionsgrenzen. In: 26. VDI/VW-Gemeinschaftstagung Fahrerassistenz und Integrierte Sicherheit, 6./7. Oktober 2010 in Wolfsburg (2010)], [ Daimler AG: Mercedes-Benz präsentiert in Genf Limousine und Coupé der neuen E-Klasse (2009)] ) How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

6 Statistical Considerations (I) Poisson Distribution (independent random process) for the probability, that k events occur in case of an expected value of λ : s λ = test s perf  λ = ratio between observed test kilometers and system performance  The system performance describes the expected travel distance between two events Wachenfeld, W., Winner, H.: Die Freigabe des autonomen Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (Hrsg.) Autonomes Fahren, pp. 439-464. Springer Berlin Heidelberg (2015) How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

7 Statistical Considerations (II) Poisson Distribution (independent random process) for the probability, that k events occur in case of an expected value of λ : s λ = test s perf = ⋅  Distance factor s a s test d  Performance factor = ⋅ s a s perf perf How big is the distance factor at a confidence level of 95%? Wachenfeld, W., Winner, H.: Die Freigabe des autonomen Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (Hrsg.) Autonomes Fahren, pp. 439-464. Springer Berlin Heidelberg (2015) How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

8 Statistical Considerations (III) For a success probability of 50% and a confidence level of 95% results:  A 3 times higher travel distance is sufficient to (statistically) validate that autonomous driving is at least as safe as humans.  In order to get this with 50% probability the expected value for travel distance has to be about 4 times higher.  If the system is just as twice as good about Wachenfeld, W., Winner, H.: Die Freigabe des autonomen Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. 10 times more travel distances are needed. (Hrsg.) Autonomes Fahren, pp. 439-464. Springer Berlin Heidelberg (2015) Depending on reference accident class testing distances would exceed 100 millions km up to 10 billion km. Even Google didn‘t drive this distance yet (less than 1% of it). How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

9 STOP!!!!! For today’s vehicles (and more extreme for aircrafts) there is not any requirement for such high milage, why here? Is there a fundamental difference? How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

10 Differences between conventional and automated vehicles Transport mission Driver Environment Driving robot and vehicle Knowledge-based Behavior Navigation Road network Selected route Time schedule Guidance/ Traffic Rule-based Behavior Conducting situation Desired speed and Vehicle trajectory Steering Vehicle Accelerating Longitudinal - motion Skill-based Behavior Stabilization and Road surface Lateraldyn. Actual trajectory and speed Sensory Input Range of safe motion states Alternative routes Current approval of vehicle doesn‘t cover the yellow area (according to [Rasmussen, J.: Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models. IEEE Transactions On Systems, Man, and Cybernetics SMC- 13(3), 257–266 (1983)] and according to [Donges, E.: Fahrerverhaltensmodelle. In: Winner, Hakuli, Wolf (eds.) Handbuch Fahrerassistenzsysteme, pp. 15–23 (2011)]) How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

11 Approval Trap With today´s method and approaches, the approval of autonomous vehicles is not imaginable! ? Autonomous Driving What next?  Abandon the development of autonomous vehicles?  Find ways out; disarm the trap! How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

12 Lack of Testing Knowledge for Driving Function We do not know …  the representative worst case test cases,  the metrics for identification of critical situations,  the environmental influence on perception,  how the behavior can be tested as robust and safe,  whether the simulation models for MiL, SiL, HiL, ViL are valid and how to validate,  how representative the simulation has to be for approval purpose. How can we gain that missing knowledge? How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015

13 Evoluationary Approach to Gain Testing Knowledge based on Functional Evolution Functional evolution  Predecessor function gains testing knowledge for successor function  Data bases of situations, remarkable road parts, of sensor raw signals  Establishing test cases (for real and virtual tests)  Virtual Assessment of Automation in Field Operation (VAAFO) can evaluate emulated behavior without danger. (Source [Wachenfeld, W., Winner, H.: Virtual Assessment of Automation in Field Operation – A New Runtime Validation Method, FAS Workshop in Walting 2015])  Makes simulation models more realistic.  Generates statistical data for risk assessment How does the functional evolution look like?  It depends on the Use Case of Autonomous Driving (Source [Wachenfeld, W., Winner, H., Gerdes, C., Lenz, B., Maurer, M., Beiker, S.A., Fraedrich, E., Winkle, T.: Use-Cases des autonomen Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (eds.) Autonomes Fahren, pp. 9-37. Springer Berlin Heidelberg (2015)]) How to Address the Approval Trap for Autonomous Vehicles | Prof. H. Winner| Dagstuhl | November 2015