S9169 Augmented Reality Solution for Advanced Driver Assistance Sergii Bykov Technical Lead 2019-03-19
Agenda Company Introduction System Concept Perception Concept Object Detection DNN Showcase HMI Concept
Company Introduction
Company Introduction Headquarters in Munich Development centers in Eastern Europe, presence in Asia 50+ experienced and talented engineers in 4 countries 10+ years of automotive experience Know-how in core automotive domains: Vehicle Infotainment, Vehicle Sensors and Networks, Telematics, Advanced Driver Unique augmented reality in the vehicle Assistance Systems, Navigation and Maps, Collaboration with scientific groups in fields Ultimately easy and safe driving of Computer Vision and Machine Learning, unique IP and mathematical talents Full visibility of autonomous driving decisions
Representation For The Driver LCD CD scr screen Sm Smart Gl Glasse ses Alt lternativ ive, Pas ast fas ast t de developing mark arket (to (today) HUD 2D HU 2D Real eal-depth HU HUD wit ith wid ide FOV in n car ar On On goin oing de development (2 (2 yea ears) s) Today
Technology
Reference Projects Under NDA AR LCD Prototype BMW demo car AR LCD Prototype German OEM AR LCD CES Demo Under NDA Under NDA AR HUD Prototype American OEM AR HUD Prototype Shanghai OEM AR Production Project Premium OEM – ongoing
Challenges of ADAS embedded platforms • Power vs Performance – Focus on performance while presuming the low power consumption • Low latency and High response frequency – Fast responses to environment changes are crucial for working in real-time • Robustness and Quality – Ensure robustness and presume quality in difficult operating conditions – Requires a lot of verification scenarios as well as adaptive heuristics • System architecture specifics for embedded real-time – Designed for real-time requirements and portability to fit to most effective hardware platforms • Hardware and software sensor fusion – Fuse available data sources (sensors, maps, etc.) for robustness and quality • Big data analysis – Huge amount of data should be stored and used for development and testing • In- and Off-field automated testing – Adaptive heuristics development – System validation – Collecting special cases
Challenges of ADAS machine learning • Machine Learning needs large volumes of quality data – Real need to ensure greater stability and accuracy in ML – High volumes of data might not be available for some tasks, limiting ML’s adoption • AI vs Expectations – Understanding the limits of technology – Address expectations of replacing human jobs • Becoming production-ready – Transition from modeling to releasing production-grade AI solutions • Current ML doesn’t understand context well – Increased demand for real-time local data analysis – A need to quickly retrain ML models to understand new data • Machine Learning security – Addressing security concerns such as informational integrity
System Concept
Apostera Approach – High Level & Highlights • Hardware and sensors agnostic • Confidence estimation of fusion/visualization • Real-time with low resource consumption • Latency compensation and prediction model – Pitch, roll, low- and high-frequency • Configurable design for different OEMs • Configurable logic requirements (including models and regions) – User interface logic considers confidence or probability of input data – Considers the dynamic environment and objects occlusion logic • Integration with different navigation systems and map formats – Compensation of map data inaccuracy – Precise relative and absolute positioning
Apostera Solution Architecture Overview
Cameras. Transport and Sensors Supplier Aptina Aptina Omnivision ADAS camera challenges Type AR0130 AR0231 OV 10635 Low Demand for algorithms reaction time Resolution pixel 1280x960 1928x1208 1280x800 Resolving data source synchronization issue latency Dynamic dB 115 (HDR) 120(HDR) 115(HDR) Small Demand for increasing number of ADAS sensors Increasingly space constrained footprint V/L- Response 5.48 - 3.65 sec Reduced heat improves image quality & reliability Low power Battery applications Frames fps 60 40 30 High Harsh environment GS/ER Shutter Type ERS ERS ERS Reliability Passenger and industrial vehicles S Sensor optical Inch 1/3” 1/2.7” 1/2.7” format (“) IP / ETH AVB / GMSL transport comparison Pixel size µm 3.75 3 4.2 Frame ~105ms encoder IP transmit decoder exposure Parallel Interface MIPI CSI2 Parallel DVP RGB 33 ms 1 ms 70 ms 1 ms Frame Encoder ETH AVB decoder Application ADAS ADAS ADAS ~37ms exposure 33 ms 1 ms 2 ms 1 ms Operation °C -40...+105 -40...+105 -40...+105 temp. line exposure serializer LVDS deserializer ~33ms Table – camera sensors comparison 45 μ s 15 μ s (per line)
Perception Concept
Sensor Fusion. Data Inference Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models with different chassis geometries and steering wheel models/parameters. Features: Absolute and relative positioning Dead reckoning Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors Fusion with navigation data Rear movements support Complex steering wheel models identification. Ability to integrate with provided models GPS errors correction Stability and robustness against complex conditions – tunnels, urban canyons
Sensor Fusion. Advanced Augmented Objects Positioning Solving map accuracy problems Position clarification: • Camera motion Placing: model: • Video-based • Road model gyroscope • Vehicles detection • Positioner Component • Map data • Road model • Objects tracking
Sensor Fusion. Comparing Solutions Reference solution Apostera solution Update frequency ~4-5 Hz Update frequency ~15 Hz (+extrapolation with any fps)
Lane Detection. Adaptability and Confidence • Low level invariant features – Single camera – Stereo data – Point clouds • Structural analysis • Probabilistic models – Real-world features – Physical objects – 3D scene reconstruction – Road situation • 3D space scene fusion (different sensors input) • Backward knowledge propagation from environment model
Ongoing work. More detection classes • Road object classes extension (without a loss of quality for existing classes) – Adding traffic signs recognition (detector + classifier) – Adding traffic lights recognition
Ongoing work. Drivable area detection • Drivable area detection using semantic segmentation • Model is inspired by Squeeze-net and U-Net. • Current performance (Jetson TX2): – Input size: 640x320 (lowres) – Inference speed: 75 ms/frame
Object Detection DNN Showcase
Object detection DNNs. Speed vs Accuracy Figure – Accuracy (mAP) vs inference time of different meta architecture / feature extractor combinations for MS COCO dataset
Single Shot Multibox Detector • Discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location • Generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape • Combines predictions from multiple feature maps with different resolutions to handle various sizes • Simple relative to methods that require object proposals, eliminates proposal generation and subsequent pixel or feature resampling stages, encapsulates all computation in a single network Figure – SSD model architecture
MobileNet as a Feature Extractor • Depth wise separable convolutions to build light weight deep neural networks • Two global hyper parameters to adjust between latency and accuracy • Solid performance compared to other popular models on ImageNet classification • Effective across a wide range of applications and use cases: – object detection – fine grain classification – face attributes – large scale geo-localization Figure - Depth wise separable convolution block and MobileNet architecture
SSD-MobileNet Qualities • Speed vs Accuracy: – SSD with MobileNet has the highest mAP among the models targeted for real-time processing • Feature extractor: – The accuracy of the feature extractor impacts the detector accuracy, but it is less significant with SSD. • Object size: – For large objects, SSD performs pretty well even with a simple extractor. SSD can even match other detectors’ accuracies using better extractor. But SSD performs worse on small objects compared to other methods. • Input image resolution – Higher resolution improves object detection for small objects significantly while also helping large objects. Decreasing resolution by 2x in both dimensions lowers accuracy, but with 3x reduced inference time. • Memory usage – MobileNet has the smallest RAM footprint. It requires less than 1Gb (total) memory.
SSD-MobileNet Detection Quality • Input size: 640x360 • Detection quality for classes (AP@0.5IOU): – Light vehicle – 0.52 – Truck/bus – 0.36 – Cyclist/motorcyclist – 0.255 – Pedestrian – 0.29
Recommend
More recommend
