3D Vision Torsten Sattler and Martin Oswald Spring 2018
3D Vision • Understanding geometric relations • between images and the 3D world • between images • Obtaining 3D information describing our 3D world • from images • from dedicated sensors
3D Vision • Extremely important in robotics and AR / VR • Visual navigation • Sensing / mapping the environment • Obstacle detection, … • Many further application areas • A few examples …
Google Tango (officially discontinued, lives on as ARCore)
Google Tango
Image-Based Localization
Geo-Tagging Holiday Photos (Li et al. ECCV 2012)
Augmented Reality (Middelberg et al. ECCV 2014)
Large-Scale Structure-from-Motion Video credit: Johannes Schönberger
Virtual Tourism
3D Urban Modeling UNC/UKY UrbanScape project
3D Urban Modeling
Mobile Phone 3D Scanner
Mobile Phone 3D Scanner
Self-Driving Cars
Self-Driving Cars
Self-Driving Cars
Micro Aerial Vehicles
Mixed Reality Microsoft HoloLens
Virtual Reality
Raw Kinect Output: Color + Depth http://grouplab.cpsc.ucalgary.ca/cookbook/index.php/Technologies/Kinect
Human-Machine Interface
3D Video with Kinect
Autonomous Micro-Helicopter Navigation Use Kinect to map out obstacles and avoid collisions
Dynamic Reconstruction
Performance Capture
Performance Capture (Oswald et al. ECCV 14)
Performance Capture
Motion Capture
Interactive 3D Modeling (Sinha et al. Siggraph Asia 08) collaboration with Microsoft Research (and licensed to MS)
Scanning Industrial Sites as-build 3D model of off-shore oil platform
Scanning Cultural Heritage
Cultural Heritage Stanford ’ s Digital Michelangelo Digital archive Art historic studies
Archaeology accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)
Forensics • Crime scene recording and analysis
Forensics
Sports
Surgery
3D Vision Course Team Martin Oswald Nikolay Savinov Peidong Liu Torsten Sattler CNB G103.2 CAB G 81.1 CAB G 84.2 CNB 104 peidong.liu@inf.ethz.ch torsten.sattler@inf.ethz.ch martin.oswald@inf.ethz.ch nikolay.savinov@inf.ethz.ch Katarina Tóthóva Johannes Schönberger Federico Camposeco CAB G 102.2 CAB G 85.1 CAB G 86.3 katarina.tothova@inf.ethz.ch jsch@inf.ethz.ch federico.camposeco@inf.ethz.ch
Course Objectives • To understand the concepts that relate images to the 3D world and images to other images • Explore the state of the art in 3D vision • Implement a 3D vision system/algorithm
Learning Approach • Introductory lectures: • Cover basic 3D vision concepts and approaches. • Further lectures: • Short introduction to topic • Paper presentations ( you ) (seminal papers and state of the art, related to your projects) • 3D vision project: • Choose topic, define scope (by week 4) • Implement algorithm/system • Presentation/demo and paper report Grade distribution • Paper presentation & discussions: 25% • 3D vision project & report: 75%
Materials Slides and more http://www.cvg.ethz.ch/teaching/3dvision/ Also check out on-line “shape-from-video” tutorial: http://www.cs.unc.edu/~marc/tutorial.pdf http://www.cs.unc.edu/~marc/tutorial/ Textbooks: • Hartley & Zisserman, Multiple View Geometry • Szeliski, Computer Vision: Algorithms and Applications
Schedule Feb 19 Introduction Feb 26 Geometry, Camera Model, Calibration Mar 5 Features, Tracking / Matching Mar 12 Project Proposals by Students Mar 19 Structure from Motion (SfM) + papers Mar 26 Dense Correspondence (stereo / optical flow) + papers Apr 2 Bundle Adjustment & SLAM + papers Apr 9 Student Midterm Presentations Arp16 Easter break Apr 23 Multi-View Stereo & Volumetric Modeling + papers Whitsundite Apr 30 May 7 3D Modeling with Depth Sensors + papers May 14 3D Scene Understanding + papers May 21 4D Video & Dynamic Scenes + papers May 28 Student Project Demo Day = Final Presentations
Fast Forward • Quick overview of what is coming…
Camera Models and Geometry Pinhole camera or Geometric transformations in 2D and 3D
Camera Calibration • Know 2D/3D correspondences, compute projection matrix also radial distortion (non-linear)
Feature Tracking and Matching Harris corners, KLT features, SIFT features key concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes
3D from Images L 2 m 1 C 1 M? M L 1 Triangulation - calibration m 2 l 2 - correspondences C 2
Epipolar Geometry Fundamental matrix Essential matrix Also how to robustly compute from images
Structure from Motion Initialize Motion Initialize Structure (P 1 ,P 2 compatibel with F) (minimize reprojection error) Extend motion Extend structure (compute pose through matches (Initialize new structure, seen in 2 or more previous views) refine existing structure)
Visual SLAM • Visual Simultaneous Navigation and Mapping (Clipp et al. ICCV’09)
Stereo and Rectification Warp images to simplify epipolar geometry Compute correspondences for all pixels
Multi-View Stereo
Joint 3D Reconstruction and Class Segmentation (Haene et al CVPR13) reconstruction only (isotropic smoothness prior) joint reconstruction and segmentation ■ Building (ground, building, vegetation, stuff) ■ Ground ■ Vegetation ■ Clutter
Structured Light • Projector = camera • Use specific patterns to obtain correspondences
Papers and Discussion • Will cover recent state of the art • Each student team will present a paper (5min per team member), followed by discussion • “Adversary” to lead the discussion • Papers will be related to projects/topics • Will distribute papers later (depending on chosen projects)
Projects and reports • Project on 3D Vision-related topic • Implement algorithm / system • Evaluate it • Write a report about it • 3 Presentations / Demos: • Project Proposal Presentation (week 4) • Midterm Presentation (week 8) • Project Demos (week 15) • Ideally: Groups of 3 students
Course project example: Build your own 3D scanner! Example: Bouguet ICCV’98
Project Topics
DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks Goal: The goal is to implement a deep recurrent convolutional neural network for end-to-end visual odometry [1] Description: Most of existing VO algorithms are developed under a standard pipeline including feature extraction, feature matching, motion estimation, local optimization, etc. Although some of them have demonstrated superior performance, they usually need to be carefully designed and specifically fine-tuned to work well in different environments. Some prior knowledge is also required to recover an absolute scale for monocular VO. This project is to implement a novel end-to- end framework for monocular VO by using deep Recurrent Convolutional Neural Networks (RCNNs). Since it is trained and deployed in an end-to-end manner, it infers poses directly from a sequence of raw RGB images (videos) without adopting any module in the conventional VO pipeline. Based on the RCNNs, it not only automatically learns effective feature representation for the VO problem through Convolutional Neural Networks, but also implicitly models sequential dynamics and relations using deep Recurrent Neural Networks. Extensive experiments on the KITTI VO dataset show competitive performance to state-of-the-art methods, verifying that the end-to-end Deep Learning technique can be a viable complement to the traditional VO systems. [1] Wang et. al., DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks, ICRA 2017 Peidong Liu, CNB D102 peidong.liu@inf.ethz.ch Recommended : Python and prior knowledge in machine learning
Recommend
More recommend
