360-Degree 3D Ground Surface Reconstruction Using a Single Rotating - PowerPoint PPT Presentation

360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera Kouma Motooka † , Shigeki Sugimoto † , Masatoshi Okutomi † Takeshi Shima ‡ † Tokyo Institute of Technology ‡ Hitachi Ltd. IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Introduction • Backgrounds – 360-degree surrounding ground surfaces. • Required for heavy machineries in construction sites. • Related works ・ Hard to calibrate ・ Expensive Catadioptric stereo camera [18] Laser scanner [18] M. Schonbein , A. Geiger. “Omnidirectional 3D reconstruction in augmented Manhattan worlds.” IROS2014 IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 2

Objective To robustly and precisely reconstruct a 360-degree 3D ground surface from the images captured by a single rotating camera. Camera positions Camera rotation 3D ground surface Single rotating camera: easy to calibrate, low cost. Robust and precise Reconstruction: via effective combination of feature-based and pixel-based methods. IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 3

Overview Robust initial surface estimation Surface refinement by fitting to 3D points of SfM by minimizing variances of pixels Ground coordinate Regular mesh system Refined surface Surface fitted to 3D Points IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 4

Robust initial surface estimation Robust initial surface estimation by fitting to 3D points of SfM Ground coordinate Regular mesh system Surface fitted to 3D Points IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 5

Overview Robust initial surface estimation Surface refinement by fitting to 3D points of SfM by minimizing variances of pixels Ground coordinate Regular mesh system Refined surface Surface fitted to 3D Points IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 6

Surface refinement The heights of each vertex Data term Smoothness term Cost function: 𝐷 𝑄 𝒜 = 𝐷 𝑄𝐸 𝒜 + 𝛽 𝑄𝑇 𝐷 𝑄𝑇 (𝒜) 1 2 Data term : 𝐽 𝑜 𝑡 𝒒 𝑜 𝑡 𝒚 𝑡 𝒚 ′ 𝑡 , 𝒜 − 𝐷 𝑄𝐸 𝒜 = 𝐽 𝑜 𝑡 (𝒜) 𝑜 𝑡 ′ 𝑜 𝑡 𝒚 𝑡 The variance of the pixel values 𝒚′ 𝑡 : Sample point on 𝑦 - 𝑧 plane 𝒒 𝑜 𝑡 𝒚 𝑡 : Point on surface 𝒒 𝑜 𝑡 : Point on image 𝑨 𝐽 𝑜 𝑡 : Image which observes 𝒚 𝑡 𝑧 𝒚 𝑡 𝐽 𝑜 𝑡 (𝒜) : Average of the pixel values 𝑜 𝑡 : Number of the image 𝐽 𝑜 𝑡 𝑦 Ground 𝒚′ 𝑡 coordinate system 𝑦 - 𝑧 plane IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 7

Outlier removal Outlier removal Same 3D point on the surface Moving shadow Outlier pixel Detect outlier pixels of the surface point 𝒚 𝑡 by taking the median value MED(𝐽 𝑜 𝑡 ) 𝐽 𝑜 𝑡 is outlier pixel 𝐽 𝑜 𝑡 𝒒 𝑜 𝑡 (𝒚 𝑡 (𝒜)) − MED(𝐽 𝑜 𝑡 ) > 𝜐 (𝑢ℎ𝑠𝑓𝑡ℎ𝑝𝑚𝑒) Minimize the cost function 𝐷 𝑄𝑇 (𝒜) using only inlier pixels IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 8

Implementation Details Pre-defined sample points Hierarchical meshing • Defined on the x-y plane using pixel- interval • Dense in near, sparse in far. (In 4 or 5 pyramid levels) IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 9

Experimental results: Synthetic images 16m 16m Setting 𝒜 Rotating camera 1m 45 ° Ground truth surface 2.0m Input image (640 × 480 pixels) 𝒚 𝒛 Ground coordinate system ・ The camera rotated 10 degrees per frame capturing 36 images IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 10

Estimated surfaces 16m 16m Ground truth surface Initial surface (SfM + surface fitting) Final (proposed method) SfM + PMVS [5] + surface fitting [5] Y. Furukawa and J. Ponce. Accurate, “Accurate, Dense, and Robust Multi - View Stereopsis” PAMI2010 IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 11

Error maps 16m 16m Ground truth surface Initial surface (SfM + surface fitting) Final (proposed method) SfM + PMVS [5] + surface fitting 10cm 0 Z error IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 12

RMSE in distances 12 SfM Initial 10 SfM+PMVS PMVS Final (Proposed) Proposed 8 RMSE [cm] 6 4 2 0 2 ～ 4m 4 ～ 6m 6 ～ 8m 8 ～ 10m 10m ～ Distance from the rotation axis IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 11

Real scene ① Scene Setting 1.2m Camera Rotation Input image (640 × 480 pixels) 2.0m ・ The camera rotated 10 degrees per frame capturing 36 images IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 12

Real scene ① : Estimated surface Scene Camera position 12m 12m Result of 3D ground surface generation Textured surface （ Grid size : 12.5cm ） IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd.

Real scene ② Input image (640 × 480 pixels) IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 16

Real scene ② : Comparison Without outlier removal With outlier removal With outlier removal （ surface ） （ surface + texture ） （ surface ） IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 17

Conclusion • 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera – Propose the method for robustly and precisely reconstructing a 3D ground surface – Improve the stability by outlier pixels removal • Future work – Acceleration by optimizing implementation IROS2015: 360-Degree 3D Ground Surface Reconstruction Using a Single Rotating Camera, Tokyo Institute of Technology, Hitachi Ltd. 18