hands on intro to developing vision and lidar classifier
play

Hands-on Intro To Developing Vision and LiDAR Classifier Formula - PowerPoint PPT Presentation

Feb 15, 2023 •248 likes •869 views

Hands-on Intro To Developing Vision and LiDAR Classifier Formula Student Driverless Workshop powered by Introduction Sibo Zhu Zhijian Liu Haotian Tang Perception Lead at Perception Lead at Perception Lead at MIT Driverless

  1. Hands-on Intro To Developing Vision and LiDAR Classifier Formula Student Driverless Workshop powered by

  2. Introduction Sibo Zhu Zhijian Liu Haotian Tang ▪ Perception Lead at ▪ Perception Lead at ▪ Perception Lead at MIT Driverless MIT Driverless MIT Driverless ▪ Research Assistant at ▪ PhD student at MIT ▪ PhD student at MIT MIT HAN Lab HAN Lab HAN Lab https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 2

  3. Introduction https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 3

  4. Vision Perception Task Wide Angle Wide Angle VIO VIO Camera Camera Camera Camera Stereovision Pair Stereovision Pair https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 4

  5. System Requirement ▪ Latency ▪ Maximum view-to-actuation time for emergency stop from top speed during an acceleration run ▪ Mapping Accuracy ▪ Driven by downstream mapper ▪ Horizontal Field-of-View (FOV) ▪ perceive landmarks on the inside apex of a hairpin turn ▪ Look-ahead Distance ▪ depends on the full-stack-latency and vehicle deceleration rate https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 5

  6. Depth Estimation Step2 Step3 ` ` ` Mono Step 1 ` ` ` Ankit. et al. IV’19 Keypoints Detection Perspective-n-Point (PnP) Step2 Stereo Object Detection + Stereo Matching Algorithm Al https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 6

  7. Hands-on Tutorial To Train Your Own Cone Detection Network colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 7

  8. Colab Tutorial ： colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 8

  9. Software Design - Keypoints Detection ▪ Detects seven keypoints on each YOLOv3 detection ▪ A residual NN that leverages the geometric relationship between keypoints ▪ Seven detected key points will be then used in a Perspective-n-Point (PnP) to get depth https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 9

  10. Hands-on Tutorial To Train Your Own Keypoints Detection Network colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 10

  11. Colab Tutorial ： colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 11

  12. Validation - Accuracy https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 12

  13. Validation - Latency Open Sourced here: github.com/cv-core https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 13

  14. Beyond This Computer Vision Tutorial… ▪ Change the object detection backbone from YOLOv3 to YOLOv4/ EfficientNet/etc ▪ Adding temporal information for more stable and accurate detection ▪ Temporal Shift Module: hanlab.mit.edu/projects/tsm/ ▪ Inference with TensorRT in C++ ▪ Open sourced here: github.com/cv-core ▪ Prune the full YOLO architecture for cone detection task ▪ Quantization (int8) for even faster inference https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 14

  15. LiDAR Perception https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 15

  16. Thank you for your attention https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 16

  17. Hands-on Intro to Developing Vision and LiDAR Classifiers Formula Student Driverless Workshop powered by

  18. Speaker Introduction Sibo Zhu Zhijian Liu Haotian Tang ▪ RA at MIT HAN Lab ▪ PhD student at MIT ▪ PhD student at MIT ▪ Perception Lead at ▪ Perception Lead at ▪ Perception Lead at MIT Driverless MIT Driverless MIT Driverless https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  19. LiDAR Perception Song Han Feb 22, 2018 3D LiDAR Sensor 3D Point Cloud 500k+ points: (x, y, z, intensity) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  20. LiDAR Perception Task Velodyne 32C LiDAR https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  21. Autonomous Racing Vehicle: Objectives Low Latency High Accuracy (Drive Faster) (Prevent Collisions) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  22. Autonomous Racing Vehicle: Challenges Self-Driving Cars A whole trunk of computers! We need more efficient algorithms that do not consume intensive computations. https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  23. Autonomous Racing Vehicle: Challenges Fewer Laser Fewer Longer Distance Rings on Objects Laser Points Too many Laser Too many Shorter Distance Rings on Objects Laser Points https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  24. Efficient LiDAR Perception: Bottleneck Bandwidth (GB/s) Sequential Memory Access 668 8 20x slower 167 30 Random Memory Access 8 Mult and Add SRAM MemoryDRAM Memory Off-chip DRAM access is much Random memory access is more expensive than inefficient due to the potential arithmetic operation! bank conflicts! https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  25. Voxel-Based Models: Cubically-Growing Memory 8 0 128 x 128 x 128 resolution 83 GB (Titan XP x 7 ) GPU Memory (GB) 7% information loss 6 0 4 64 x 64 x 64 resolution 0 11 GB (Titan XP x 1 ) 42% information loss 2 0 3D ShapeNets [CVPR’15] 0 VoxNet [IROS’15] 20 40 60 80 100 120 3D U-Net [MICCAI’16] Voxel Resolution https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  26. Point-Based Models: Sparsity Overheads DGCNN PointCNN SpiderCNN Ours 95,1 57,4 51,8 51,5 45,3 36,3 27,0 Runtime (%) 15,6 12,2 PointNet [CVPR’17] 4,9 2,9 0,0 PointCNN [NeurIPS’18] Irregular Access Dynamic Kernel Actual Computation DGCNN [SIGGRAPH’19] https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  27. Point-Voxel Convolution (PVConv) Voxelize Convolve Devoxelize Fuse Normalize Multi-Layer Perceptron https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  28. Point-Voxel Convolution (PVConv) Multi-Layer Perceptron Point-Based Feature Transformation (Fine-Grained) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  29. Point-Voxel Convolution (PVConv) Voxel-Based Feature Aggregation (Coarse-Grained) Voxelize Convolve Devoxelize https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  30. Point-Voxel Convolution (PVConv) Voxel-Based Feature Aggregation (Coarse-Grained) Voxelize Convolve Devoxelize Fuse Normalize Multi-Layer Perceptron Point-Based Feature Transformation (Fine-Grained) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  31. Low Resolution with Constrained Memory Original Scene Downsampled Scene https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  32. Sparse Point-Voxel Convolution (SPVConv) Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  33. Sparse Point-Voxel Convolution (SPVConv) Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  34. Designing Efficient 3D Modules (SPVConv) trunk traffic sign trunk person cyclist person https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  35. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  36. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  37. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

Recommend

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI MILI: 2-channels UV eye- safe scanning lidar RALI RALI: 6-channels multiwavelength lidar FLUORES FLUORES: fluorescence lidar OLI

278 views • 10 slides
Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js

Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js

Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js Interactive Web Pages Cross-platform libraries Client-server integration Wrap Up Intro to Scala.js Intro to Scala.js

547 views • 37 slides
CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization,

CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization,

CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization, Momentum, Regression, Multi-label Classification / Tagging Previous Class Softmax Classifier Inference vs Training Gradient Descent (GD)

564 views • 29 slides
June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of

June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of

The multiple uses of LiDAR CDA Basin WAG June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of Light x Time of Flight)/2 3 Correcting our Existing GIS Databases 4 Mining and Geology 5 Land Form

476 views • 28 slides
Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

CS6501: Deep Learning for Visual Recognition Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning? Supervised Learning: Classification with K-nearest neighbors Unsupervised Learning: Clustering with

1.36k views • 91 slides
Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research

Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research

Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research hands Prosthetics Design issues Kinematics Compliance Sensing Actuation Control Robustness Evaluation Discussion Hands of the 80s Hirose

238 views • 22 slides
LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees

LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees

LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees SEPTEMBER 5 - 7, 2018 Terrestrial Laser Airborne LIDAR Unmanned Aerial Platforms Scanning Systems LiDAR Types Applications Platforms

362 views • 22 slides
The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May

The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May

The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May 2019 cas.uts.edu.au LIDAR - INERTIAL 3D- Lidar IMU (300kHz points) Lidar Depth (10Hz) IMU Acceleration, velocity (100Hz) Sparse

438 views • 23 slides
Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR

Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR

Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR Scanning Overview Scanning background Applications Strengths Weaknesses Scanning Examples Fisher Associates LiDAR = light

610 views • 20 slides
Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution

Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution

Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution Information The Role of Product Tracing: t d today and tomorrow d t Caitlin Hickey Caitlin Hickey Jennifer McEntire Jennifer McEntire Institute of

174 views • 15 slides
Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands

Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands

Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands on intro to CADRE Program (Mat Hutchinson) overview Interactive demo with packages and notebooks (Filipi Silva) CADRE fellow presentation

878 views • 65 slides
What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the

What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the

What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the LiDAR (from the canopy) 18 07/12/2018 LiDAR data (25cm) following initial processing (vegetation removal) Historic England (2018) Using

319 views • 30 slides
Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic

Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic

Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic fatalities and serious injuries in Washington DC by 2024 through the use of data, education, enforcement, and engineering. Vision Zero work is

454 views • 7 slides
Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System

Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System

Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System Administration DeCal 2012-08-27 1 / 21 The connection Two of the most The connection Introduction famous products of Diving into the shell

585 views • 21 slides
S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES

S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES

S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES Vision Everyone in missions Mission Serving hope by sharing Gods love and message unequivocally to all peoples Values Relationships

665 views • 25 slides
Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Outline Background and Motivation Classifier Ensembling Genetic Algorithms Proposed Method of Classifier Ensemble Feature Set Used Experimental Results Conclusions Future Works Maximum Entropy Classifier Ensembling using Ge- netic

749 views • 36 slides
GPU WORKSHOP University of Maryland 1 Intro to GPU Computing 2 OpenACC with hands-on AGENDA 3

GPU WORKSHOP University of Maryland 1 Intro to GPU Computing 2 OpenACC with hands-on AGENDA 3

GPU WORKSHOP University of Maryland 1 Intro to GPU Computing 2 OpenACC with hands-on AGENDA 3 CUDA C/C++ with hands-on 4 5 2 Parallel programming Why do you care? 3 The world IS parallel Accelerator Programming Why do you Care?

1.08k views • 55 slides
LiDAR Teach-In OSRAM Licht AG | June 20, 2018 | Munich Light is OSRAM Agenda Introduction

LiDAR Teach-In OSRAM Licht AG | June 20, 2018 | Munich Light is OSRAM Agenda Introduction

www.osram.com LiDAR Teach-In OSRAM Licht AG | June 20, 2018 | Munich Light is OSRAM Agenda Introduction Autonomous driving LIDAR technology deep-dive LiDAR@OS: Emitter technologies Outlook OSRAM Licht AG 2 LiDAR Tech

798 views • 22 slides
THE PROGRESSION OF LIDAR Tuesday November 12, 2013 Historical LiDAR data Aerial Platforms

THE PROGRESSION OF LIDAR Tuesday November 12, 2013 Historical LiDAR data Aerial Platforms

NYGeoCon 2013 Saratoga Springs, NY THE PROGRESSION OF LIDAR Tuesday November 12, 2013 Historical LiDAR data Aerial Platforms Ground based. Project areas cover several hundred square miles. Typical point densities of 1

385 views • 13 slides
No hands! + Using data to automate performance marketing Intro to Western Union Hundreds of

No hands! + Using data to automate performance marketing Intro to Western Union Hundreds of

Look mum, No hands! + Using data to automate performance marketing Intro to Western Union Hundreds of thousands of In over 200+ countries We moved more than agent locations around and territories $300bn in principal the globe around the

169 views • 14 slides
Technology Mial Warren VP of Technology October 22, 2019 Outline Introduction to ADAS and

Technology Mial Warren VP of Technology October 22, 2019 Outline Introduction to ADAS and

Navigating Automotive LIDAR Technology Mial Warren VP of Technology October 22, 2019 Outline Introduction to ADAS and LIDAR for automotive use Brief history of LIDAR for autonomous driving Why LIDAR? LIDAR requirements

603 views • 29 slides
Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro Computer Vision Announcements Midterm Percentage Score Out of Mean 67 52. 78 pts. Median 62 48 High 96 75 Low 24 18 CSE152,

315 views • 17 slides
HEAVENS HANDS HISTORY http://www.hhcsny.org/ Heavens Hands started in 2000 Grown from a

HEAVENS HANDS HISTORY http://www.hhcsny.org/ Heavens Hands started in 2000 Grown from a

HEAVENS HANDS HISTORY http://www.hhcsny.org/ Heavens Hands started in 2000 Grown from a staff of 2 to over 200 Support more than 900 individuals and families CEO: Lorenzo Brown COO: Mary Knox CFO: Eugene Brandon MISSION & VISION

514 views • 12 slides
PyMTL/Pydgin Tutorial Schedule 8:30am 8:50am Virtual Machine Installation and Setup 8:50am

PyMTL/Pydgin Tutorial Schedule 8:30am 8:50am Virtual Machine Installation and Setup 8:50am

Presentation Presentation Presentation Hands-On Presentation Hands-On Hands-On Overview Pydgin Intro GCD Instr PyMTL Intro Max/RegIncr ML Modeling GCD Unit PyMTL/Pydgin Tutorial Schedule 8:30am 8:50am Virtual Machine

487 views • 7 slides

More recommend