User-assisted Segmentation and 3D Motion Tracking Michael Fleder - PowerPoint PPT Presentation

User-assisted Segmentation and 3D Motion Tracking Michael Fleder Sudeep Pillai Jeremy Scott

3D Object Tracking ● Virtual reality and animation ● Imitation in robotics ● Autonomous driving ● Augmented reality

Motivation Want low-cost tracking with no instrumentation

Method Overview User-assisted Segmentation

Method Overview Iterative Closest Point (ICP) for 3D tracking

Method Overview Color Histogram 2D tracking to seed ICP

DEMO

Preliminary Experiments / Results ● FAST features for saliency detection ● ORB / SURF / SIFT feature descriptor extractor ● SURF/ SIFT ● Robust ● Lacking real-time capabilities ● ORB (Oriented FAST and Rotated BRIEF) ● Runs in real-time ● Produces a lot of false positives

Preliminary Experiments / Results DEMO

Preliminary Experiments / Results ● Limitations ● Kinect RGB Sensor – Low resolution – Signal-to-noise ratio not particularly good ● Feature detection – Doesn't cope well with motion blur – Actively learn new features of the object – Correspondence requires rigid object tracking

Appearance-based tracking ● Appearance model + Temporal coherence ● Histogram appearance models – Base color representative over multiple views ● Robust to perspective views ● Don't have to actively learn new features ● Sufficient to provide object localization belief – Relatively faster to compute ● Mean-Shift tracking – Robust to motion blur – Eliminates false positives that have same color distribution

Two approaches ● Create a likelihood image, with pixels weighted by similarity to the desired color ● Best for uni-colored objects ● Fails when there is a color distribution ● Represent color distribution with a histogram ● Use mean-shift to find region that has most similar distribution of colors

Comparing color distributions ● Bhattacharya Distance/Coefficient – Measure of the amount of overlap between two statistical samples i.e. between histograms – Imposes a metric structure – Invariant to object size (number of pixels) – Valid for arbitrary distributions

Mean-Shift Algorithm ● Finding modes in a set of data samples, manifesting an underlying PDF in R N ● Feature space can be color, scale or saliency

Mean-Shift Algorithm

Mean-Shift Algorithm

Mean-Shift Algorithm

Mean-Shift Object Tracking Target Representation

Mean-Shift Object Tracking PDF Representation

First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point 3 Given: Two point-sets A (object to track), B (new point cloud) ⊆ R Goal: Find a one-to-one matching function μ : A → B that minimizes the root mean squared distance (RMSD) between A and B Incorporating rotation R and translation t:

First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point ICP(A,B) : // A,B point sets. R = rotation matrix. t = translation vector. //Find μ one-to-one matching function. 1. Initialize R = I (the identity matrix), t = 0. 2. Matching Step: Given R and t, compute optimal μ by finding min μ RMSD(A, B, μ). 3. Transformation Step: Given μ, compute optimal R and t by finding min R,t RMSD(RA − t, B, μ). 4. Go to step 2 unless μ is unchanged.

First Attempt: Tracking using 3D Only (No RGB) Iterative Closest Point Problems: (1) Can’t handle large displacement between frames (2) Computationally intense (3) Ambiguous for geometrically-symmetric objects Benefits: (1) Resolve 3D orientation (2) Accurate for small-displacements

Third Pass: Tracking using 2D and 3D Tracking Benefits: 2D (Histogram): (1) Handles large displacements between frames (2) Accurate small-patch tracking (3) Fast 3D (ICP): (1) Accurate for small-displacements (2) Resolves 3D orientation Combined: Seed 3D ICP with result of 2D tracking

Third Pass: Tracking using 2D and 3D Initialize_Tracker (Segmented_Object, out trackedObject): (1) 2D_Segment_Points 3dTo2D(Segmented_Object) ← (2) Init_Histogram_Tracker(2D_Segment_Points) (3) trackedObject ← Segmented_Object Run_Tracker (lastKnownObject3D, latest_Image, latest_Cloud): (1) 2D_Points = run2DTracker(latest_image) (2) 3D_Search_Cloud = 2dTo3d(2D_Points) (3) lastKnownObject3D ← ICP(lastKnownObject3D, 3D_Search_Cloud)

Summary Enabling low-cost 3D object tracking on the Kinect via: 1. User-assisted segmentation 2. Combining color histogram tracker in 2D and ICP in 3D 3. Modifications to enable real-time tracking