Vijay John, Yuquan Xu , Seiichi Mita, Sm Smart t Vehicle Research Center Hossein Tehrani, Tomoyoki Oishi, Masataka Konishi, Hakusho Chin Advanced Mobil ilit ity Develo lopment Kazuhisa Ishimaru, Sakiko Nishino Res esea earch Divis isio ion 2.0
• ADAS and Automated Driving • World 3D Reconstruction • 3D Deep Sensor Fusion • Future Plan • Conclusion
Vehicle Platform ADAS App pplications ar are bo booming • Adaptive Cruise Control (ACC) • Adaptive Front Lights (AFL) • Driver Monitoring System (DMS) • Forward Collision Warning (FCW) • Intelligent Speed Adaptation (ISA) • Lane Departure Warning (LDW) • Pedestrian Detection System (PDS) • Surround -View Cameras (SVC) • Autonomous Emergency Braking (AEB) Sensors Configuration
Cameras Cam Stereo St Se Sensor Fusi usion & & Pat ath Plannin ing g / Las aser Sen ensor Per erception Contr trol Beh ehavio ior (360 deg (36 deg Sc Scene RADAR RA Gen eneratio ion Understanding) Un So Sonar GPS PS/IMU Sensors Deep Understanding of f Environment
<Process> <St Stereo Vis ision> Cal alibration Far Shift = Disparit ity far - small shift Right Left Clos lose Disparity Disp ty Calc Calculati tion (St (Stereo mat atchin ing) Far close – big shift 𝐶 . 𝑔 𝑒𝑗𝑡𝑢𝑏𝑜𝑑𝑓 = 𝑒𝑗𝑡𝑞𝑏𝑠𝑗𝑢𝑧 Far Close Dis isparity ma map De Detection Close Car Barrier 3D-Object Road
< Matching Cost > <Cost Space > Lef Left Ima Image e Right Ima Image e Eac ach pixel l has as suc uch mat atchin ing g cost cur urve, , which constructs the e Mat atchin ing Cost t Spac ace Disparity: 55 40 24 Disparity: 55 40 Image height 55 40 24 Ground Truth 0 Finding tr true disparit ity valu lue for every pixel l fr from Matchin ing Cost Space
Left Image Right Image Focus on Red pixel 0.064 Neighbors: 10 low matching cost Disparity 20 Matching cost 30 40 50 60 0.008 300 400 500 600 Matching cost curve Horizontal Axis
Exploiting the neighbors’ matching cost can be translated into Mathematical Optimization about the Sh Shortest Path th Problem Viterbi node Path cost 0.018 Viterbi 60 (optimal) 50 atching cost Disparity 40 0.012 30 Mat 20 10 0 Block 0.006 Matching 430 440 450 460 470 480 490 (wrong) Pixel Vit iterbi alg lgorithm can fin find the glo lobal optimum
VSL j , j=1,…,640 Right down to left up Viterbi direction Cost for VSL j DSL_RDLU k , k=1 ,…, K DSL_RDLU k , k=1,…,K Merge Costs Costs Average Regrouping Cost for VSL j+1 VSL j−1 VSL j VSL j+1 VSL j+2 Left up to right down Viterbi direction Huge Networks wit ith Parallel Optimization
< MPV > < SGBM > < Proposed Method(Multi-Path Viterbi) > Independently merge Cost Hierarchically merge Cost space space Viterbi→ Viterbi ← → Viterbi← Viterbi↑ Viterbi ↑ ↓ Merge Viterbi↓ Viterbi Viterbi ↖ ↗ Viterbi ↘ ↙ Viterbi Viterbi ↗ ↖ ↙ Viterbi Merge + ↘ Diagonal directions … Out: disparity Output: disparity MPV Optimize the Accumulates Information Step by Step
Blo lock Matching Dense & sm smooth th Co Conventio tional meth ethod: Multi Path Viterbi SGB SGBM
： １２８０ × ９６０ Im Image Si Size 15ms ／ Frame Calculati tion Tim Time : ： GPU GeFORCE GTX 1080 Nagoya Ur Urban Roa oad
:15ms ／ Frame Calc lculation Time :1 ： GPU GPU GeFORCE GTX 1080 Tokyo Metropolitan Highway
15ms ／ Frame Calculati tion Tim Time : ： GPU GeFORCE GTX TX 1080
Electromagnetic Range Wave: Camera ,Laser, Radar Can an Se See De Detail Can an See ee Far ar 10 ｍｍ 100 μ m 1 ｍｍ 0.1 μ m 1 μ m 10 μ m Sun Light Wave Length Sno Snow Rain Fog & Dust ~100 ／ｍ 3 ~100/c ｍ 3 Visible Light Camera Laser Electromagnetic Wave Infrared Camera Car Radar Sound Wave Car Sonar
Training a le learning fr framework for perception tasks Sen ensors Traditional l Learning Cam ameras Featu ture Extr tracti tion (HOG, Stereo St DPM etc) Las aser Perception Sensor Se Featu ture Cla lassifi ficati tion (Learning (SVM, Random Forest etc) Free Space Framework) Objects Deep le learning ( Feature Extr tractio ion + Feature Cla lassification )
• Single sensor-based learning is not robust or descriptive enough Single Sensor Camera, Lidar, Stereo Perception • Challenges (L (Learning – Environmental Variation (occlusion, Framework) illumination variation, etc.) Labels – High Inter-Class and Intra-Class Variability
There are many vehicle vari rieties with ith diff ifferent ori rientations
We have a large number of On-Road Objects We have a lot of variety of on road objects!!!!
We have the different type of road boundaries Guardrail Concrete Curb Wall Pylon Divider We have a lot of variety of Free Space Boundary!!!!
Illumination variation as observed by a monocular camera image with appearance features
• Sensor Fusion-based Cam ameras learning with Complementary Sensors Stereo St Se Sensor addresses these issues Las aser Sen ensor Fusion and Sensor Fusion Perception • Monocular Camera appearance features and (Learning (L depth features are Framework) Comple lementary Features Labels
Monocular Camera Depth th Camera Monocular Camera ⇒ Rich Depth Camera ⇒ Depth Appearance Information Information (3D Data) In Inexpensive St Stereo-based Depth th Ine Inexpensive Illumination In Ill Invariant due to Ill Illuminati tion Varia iati tion robust stereo algorithm [1] Depth th in informati tion fr from st stereo ca camera rob obust to o ill illuminati tion variati tion [1] Xu et [1] et al. al. Real eal-time St Stereo Disp sparity Qual ality Imp Improvement for Ch Chall allenging Traffic En Environments, , IV IV 20 2018
Appearance and and Depth Features are Fused wit ithin in a Deep le learnin ing Framework for Environment Perception Sensor fu Se fusion wit ith co complementa tary featu tures Deep le learning Appearance Depth th fr framework (Monocular cam (M camera) (S (Stereo Camera/Laser) Descriptive Ill Illuminati tion in invaria iant t Ap Appearance Fea eatu tures depth th featu tures 3D Environment Perception
Sensor Fusion : Raw Data Level Fusion Bad Image & Depth Free Space Detection Image + Feature Feature Extraction Object Detection Deep Features Sensor Fusion : Feature Level Fusion Feature Good Image Integration Free Space Detection Image Feature Extraction Depth Feature Object Detection Extraction Depth
Skip ip Conn onnecti tions Fea eatur ture Map ap Featu ture Ma Map U ps psampling_ Ups Upsampling_ DC1_Int DC2_Int US US_D _DC1_Int DC2 C2_Int DC1 C1_Int C1_I _Int C2_I _Int C2_I _Int C1_I _Int C2_D _Dep C1_D _Dep Fea eatur ture Map ap Con onca catenation Free Space ce Con onca catenation Outp utput C2_I _Int Int ntensi sity Inp nput ut Con onca catenation Con onca catenation C2_Dep Upsampling_ Ups Fea eatur ture Map ap Ups Upsampling_ De p DC2_De De p DC1_De DC2_Dep DC DC DC1_Dep C1_Dep Co Concatenation C2_I _Int C1_I _Int C2_Dep C1_D _Dep C2_D _Dep Featu ture Ma Map Object t Feat eature Map ap Outp utput Depth Inp nput ut Skip ip Conn onnecti tions Skip ip Conn onnecti tions Ent Entire De Dept pth Enc Encoder Feat eature Ma Maps ( m,n,n ) ) ar are tr transferred to o Free Sp Space and and Ob Object De Decoder Feature Ma Maps ( o,n,n ) ) for or Conc oncatenation ( m+o,n,n ) Entire In Ent Intensity En Encoder Feat eature Ma Maps ( m,n,n ) ) ar are tr transferred to o Free Sp Spac ace and Ob an Object De Decoder Fea eature Ma Maps ( o,n,n ) for Concatenation ( m+o,n,n )
De Dept pth Feat eature Ob Objects Feat eature Extraction Ex Detection De In Integration Dep epth Objects Obje bjects Free ee Spa Space Intensit Int ity Free Space Image Feat Ima eature Freespace Extraction Ex De Detection
Disparity Disp ty Im Imag age Objects Ob ChiNet Inten Int ensity ty Im Imag age Free Sp Spac ace • Trained with 9000 Sa Samples from Japanese Highway dataset • Manually annotated free space and objects • Trained on Keras wit ith the theano ba backend • Trained with Nvid idia ia Tita Titan X X GPU
Free ee Spac ace e Obje bjects
Implemented on GeF eForce Tita Titan X X using Keras with Theano backend
Comparison : “Intensity” vs “Intensity and Depth” Intensity image only Car detection Car not Not accurate detected Pylon not Wrong boundary detected Intensity and Disparity fusion Car, better detection Car detected Pylon detected Accurate boundary
Evaluation Result Comparison : “Intensity” vs “Intensity and Dept In Intensity im image only Pylon not detected Wrong boundary False object Intensity and Depth Fusion Pylon better boundary No false object detected
Intensity Image So Some of f Learned Im Image Feature • Edge • Vehicle Lower Part • Free Space • Free Space Depth Strong • Sky • Driving Lane Weak
Recommend
More recommend
