Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello - PowerPoint PPT Presentation

Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello (andrea.fusiello@uniud.it) -- DIEGM University of Udine Roberto Toldo, Filippo Fantini, Simone Fantoni, Luca Giona -- 3Dflow srl

www.3dflow.net 3DF Zephyr Software per la ricostruzione 3D da fotografie Sviluppato da 3Dflow srl spinoff UniUD (ex UniVR) Gestisce blocchi non strutturati Gestisce camere eterogenee con calibrazione ignota Adatto sia per la ricostruzione di piccoli oggetti che per rilievi fotogrammetrici UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

3DF Zephyr pipeline Image Image 3d photo-consistency 3d surface from acquisition orientation from images 3d photo-consistency Poisson [Kazhdan, 3DF Samantha 3DF Stasia Bolitho, and Hoppe, (structure and (multiple view 2006] motion) stereo) Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

3DF Zephyr pipeline Image Image 3d photo-consistency 3d surface from acquisition orientation from images 3d photo-consistency Poisson [Kazhdan, 3DF Samantha 3DF Stasia Bolitho, and Hoppe, (structure from (multiple view 2006] motion) stereo) Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Structure from motion Credits to R. Toldo (3Dflow s.r.l.) , R. Gherardi and M. Farenzena (U. Verona)

Structure from motion 2-­‑d ¡%e-­‑points ¡ matching ¡ object ¡3-­‑d ¡points ¡ UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Structure from motion Motion = exterior orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Orientation of a block ¤ Block: set of overlapping photographs ¤ Independent models block adjustment ¤ Create independent stereo-models with relative orientation for each pair of overlapping photographs ¤ Simultaneous transform the models into the ground coordinate system with absolute orientation (given known GCP). ¤ Bundle block adjustment ¤ Compute directly the relations between image coordinates and object coordinates, without introducing model coordinates as intermediate step. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

� � � � � � � � � � � � � � � � � Bundle block adjustment � � � � � � � � � � � � � � � � � � � � � � UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Resection-intersection cycle ¤ 1. Initialize: ¤ (a) Select two suitable photographs; ¤ (b) Solve relative orientation and form the stereo-model ( intersection ) ¤ 2. For every additional photo (following a suitable order), ¤ (a) Solve exterior orientation (resection) ¤ (c) Refine the existing model ( intersection ); ¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ This is also known in CV as sequential structure from motion (à la Bundler) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Hierarchical variation ¤ The previous method can be generalized by organizing the photographs on a tree instead of a chain . ¤ The tree is produced by hierarchical clustering the photographs according to their overlap (the overlapping relationship is indeed not a chain ) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Hierarchical variation (Samantha) ¤ 1. Form many independent stereomodels from photo pairs at the leaves of the tree ¤ 2. Traverse the tree; in each node one of these operations takes place: ¤ Grow one model by adding one photo with resection followed by intersection; ¤ Merge two independent model with absolute orientation; ¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ Note: ¤ If the tree reduces to a chain , the algorithm is a sequential SfM ; ¤ If the tree is perfectly balanced , only the Merge step is taken, and the resulting procedure resembles the IMBA. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Tree traversal Relative Orientation Exterior Orientation Absolute Orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Comments ¤ CV methods are designed for irregular and unknown block structure. Recovering block structure and tie-points matching is fundamental (see forward) ¤ CV methods do not make use of GCP but at the end ( free solution ) because they are compulsory. ¤ The gold standard is bundle adjustment, the other methods are seen as tools to get an approximate solution. ¤ The hierarchical method is more effective than the sequential one in drift containment. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Pre-processing summary ¤ Keypoint extraction ¤ Matching - broad phase: select O(n) views to be matched ¤ Matching – narrow phase: match keypoints between pair ¤ Clustering: determine processing order (can be on-line) UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Keypoints ¤ Detector : scale-space extrema of the scale-normalized Laplacian [T. Lindeberg, 1994]. SIFT is an variation on this theme. ¤ We use a 8-level scale-space and in each level the Laplacian is computed by convolution (in CUDA) with a 3 × 3 kernel. ¤ Key: multiresolution pyramid based on derivative operator. ¤ Descriptor : 128- dimensional radial descriptor based on the accumulated response of steerable derivative filters. ¤ Key: derivatives, directions histogram, UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Which images are to be matched? ¤ Recover the image graph , i.e., the graph that tells which image overlaps (or can be matched) with which other. ¤ For each key-point descriptor its approximate k nearest neighbours in feature space are computed (via ANN) ¤ A 2D histogram is then built: increment bin(i,j) whenever a keypoint of image i has a keypoint of image j in its k-neighbourhood; UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Which images are to be matched? ¤ Image graph G = (V,E) where V are views and the weighted adjacency matrix is the 2D histogram. ¤ This graph has | V | = O(n 2 ). The objective is to extract a subgraph G’ with a number of edges that is linear in n. ¤ In graph theory, a graph is k-edge-connected if it remains connected whenever fewer than k edges are removed. ¤ We devised a strategy that builds a subgraph G’ of G which is k-edge-connected by construction and has has (n-1)k = O(n) edges UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Matching ¤ Match keypoints (images connected in the graph) ¤ Validate matching with relative orientation (RANSAC) ¤ Non linear refinement of relative orientation UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Tracks ¤ From pairwise matches to tracks ¤ just a matter of bookkeeping… UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Clustering ¤ Agglomerative, hierarchical clustering problem ¤ Simple (complete, average, ward’s) linkage ¤ Distance needed: ¤ Common matches ¤ Good coverage UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

3DF Zephyr pipeline Image Image 3d photo-consistency 3d surface from acquisition orientation from images 3d photo-consistency Poisson [Kazhdan, 3DF Samantha 3DF Stasia Bolitho, and Hoppe, (structure and (multiple view 2006] motion) stereo) Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Multi-view stereo Credits to R. Toldo (3Dflow s.r.l.), and G. Vogiatzis (Aston U.)

Depth-map fusion 1. Compute depth hypotheses 2. Volumetrically fuse the depth-maps 3. Extract a 3d surface Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Computing depth maps ¤ For each pixel in the current image ¤ March along the back-projected ray of the pixel inside the bounding volume ¤ For each depth value d, project the 3D point in all the neighbouring views and compute NCC ¤ Equivalently, we work in image space on rectified images (easy to code on GPU) ¤ The mapping between epipolar lines is a 1-d homography UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Computing depth maps UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Aggregating NCC scores ¤ Several NCC profiles must be aggregated to extract the depth of the given pixel. ¤ All local maxima could be good hypotheses. ¤ How do we pick right one? Images from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

MRF estimate ¤ Voting: each local peak casts a vote (weighted by its score value) on a discrete histogram along the optical ray. ¤ Winner take-all: the most voted value is the depth of the point à can do better ¤ MRF relaxation: ¤ the k bins of the histograms with the highest score are as the candidate depths for the pixel. ¤ minimization selects the best depth among several hypotesis taking neighbourhoods into account. UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

MRF result Winner-take-all Depth map after MRF UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Merge depth maps ¤ Depth maps are lifted in 3D space to produce a photoconsistency volume, represented by an octree that accumulates the scores coming from each depth map. Image from Vogiatzis, 2010 UAV / RPAS in Italia - Modena 20-21 Febbraio 2014