Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery - PowerPoint PPT Presentation

Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery Temporal Feature Tracking for Robotic Assisted Endoscopic Surgery Haytham Elhawary, Ph.D. Philips Research North America, Briarcliff, New York, USA

Objectives of this talk • Understand the need to develop robust tracking algorithms using uncalibrated monocular endoscopes • How to choose good features to track in your image • Explore the basics of optical flow algorithms for tracking Methods to evaluate your feature tracking algorithm • • Example applications • Conclusions Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Why develop robust feature tracking algorithms? You can do lots of cool stuff with it! • Motion analysis • Mosaicing or image stitching • Track moving targets • Motion compensation • 3D reconstruction • … Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

The need for feature tracking Lab set-up of the system • IBM/JHU LARS robot 7 DOFs o (X-Y-Z stage, 3 rot, 1 insertion) Remote center of motion (RCM) o • Richard Wolf Inc endoscope Monocular, direct o Totally uncalibrated o 352 x 240 o • Chamberlain Group beating heart phantom Realistic outer surface o Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

The need for feature tracking Clinical requirements of the system • Use of a standard 2D monocular endoscope Most used endoscope in manual laparoscopic procedures o • No calibration of the endoscope Tedious technical process not apt for OR o • Allow intra-operative replacement of the endoscope Change from angled and straight lens endoscopes o • No marker placement on the heart surface Adds complexity to the procedure o Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Feature Detection “I like when a girl knows what she looks like and dresses to accentuate those features.“ Zac Efron Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track Feature Detection • There are features in an image that are more prone to be tracked successfully • Features have mathematical characteristics that fit the tracking method (optical flow) better • Several methods: Moravec corner detection, Shi and Tomasi 94 1 , and SURF based feature detector from Bay et al 2006 2 etc 1 J. Shi and C. Tomasi (June 1994). "Good Features to Track,“ 9th IEEE Conference on Computer Vision and Pattern Recognition . 2 H. Bay , et al., "SURF: Speeded Up Robust Features," in Computer Vision – ECCV 2006, vol. 3951, 2006, pp. 404-417. Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track Feature Detection Learning OpenCV, Gary Bradski & Adrian Kaehler, O’Reilly Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track Feature Detection • Good features to track method (Shi and Tomasi 94) → ( , ) I x y [ , ] = T x x y Given a point in an image the Hessian matrix at o t t t that point is:  2 2  ∂ ∂ → → ( ) ( ) I x I x   2 t t ∂ ∂ ∂ → x x y   ( ) = H x 2 2 t ∂ ∂  → →  ( ) ( ) I x I x   t t 2 ∂ ∂ ∂ x y y   The autocorrelation matrix of second order derivative images around a o small window at each point is calculated, and the eigenvalues computed. Ratio between min and max Eigen values are calculated, and if above a o threshold, the point is considered good to track. Implemented by GoodFeaturesToTrack function in OpenCV. o Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track GFT Feature Detection • Varied the ratio between max and min eigen values above which a point is considered good to track • Defined high, medium and low quality points depending on ratio Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track GFT Feature Detection Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track Feature Detection • SURF feature detector method (Bay et al, 06, 08) Speeded Up Robust Features (SURF) is a feature descriptor with 64 or 128 o dimensions SURF describes distribution of the intensity content within an interest o point neighbourhood, based on Haar wavelets (inspired by SIFT) Detects blob like structures at locations where the determinant of the o Hessian is maximum – Hessian used is covariance of second order Gaussian image Implemented in OpenCV o Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track SURF Feature Detection • Varied the threshold of the determinant above which a point is considered good to track Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track SURF Feature Detection Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Choosing good features to track SURF and GFT Feature Detection – not the same points! GFT detector SURF detector Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Feature Tracking (optical flow) “Are you stalking me? Because that would be super.“ Ryan Reynolds Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Optical flow algorithm • Optical flow allows you to detect the apparent motion of certain patterns in an image (displacement vectors) • Iterative differential method for tracking point correspondences between 2 frames Dense and sparse optical flow methods available, although • dense methods are computationally expensive Lucas-Kanade algorithm 1 (81) is one of the most popular sparse • tracking methods Dense methods include Horn-Schunck 2 (81) • 1 B. D. Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121--130 2 B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial Intelligence , vol 17, pp 185-203, 1981 Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Optical flow algorithm Lucas-Kanade method • Based on 3 assumptions: ( , , ) ( , , ) = + + + ∆ I x y t I x u y v t t Brightness constancy: o Small motions: motion is slow compared to frame rate (allows o approximating the derivative of intensity over time) Spatial coherence: neighbouring points in a scene belong to the same o surface and have similar motion 0 + + = I u I v I (1) x y t Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.

Optical flow algorithm Lucas-Kanade method • Using a window of pixels w=[w x ,w y ], (assumption 3) that move in the same manner, equation (1) can be applied for each pixel in the window to obtain u and v → Minimization problem solved with least squares = + y y w = + x x w � t y � � � t x ∑ ∑ ( ) ( ) ( ) = − + e v I x I x v + ∆ t t t t t t t = − = − x x w y y w t x t y • Implemented in OpenCV Pyramidal version of Lucas-Kanade 1 solves the problems of • large motions falling outside of the local window 1 J.-Y. Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm," Intel Corporation, Microprocessor Research Labs, unpublished. Haytham Elhawary – Temporal Feature Tracking using an Endoscope 22 nd September ,Toronto, Canada Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.