PST TP - 215 Real-Time Non-Rigid Image Registration (Image Morphing) Engine Propulsion Science and Technology, Inc. (PST) Randall Miles, Donna Yosmanovich, Michael Thorwart, Margaret Taylor GPU Technology Conference 2016 San Jose, CA April 5, 2016 1
Willow https://goo.gl/0OxGAk The film, Willow , directed by George Lucas and staring Warwick Davis, was the first example of extensive use of image morphing in film. Dennis Muren, Creative Director at Industrial Light and Magic, said the scene took weeks of computer work at the time, but could be done on a cell phone today. 2
Example: Image Morphing Michael Jackson’s music video “Black and White” is another early example of image morphing. In the video, faces of individuals of various nationalities are morphed from one to another as they dance and smile on the screen. The frame shown here capture the transition from a western male to an asian female. 3
Outline • Introduction – Who is PST? – Why are we interested in image registration? • What is image registration? – A basic definition – Examples • Solution: Image Morphing Toolkit – The library – Examples • Conclusion 4
Who is PST? (Propulsion Science and Technolgy) Scientists with a passion for • Science • Technology • and rockets! Our historical focus: modeling exhaust plumes and wakes Physics based modeling and simulation is time consuming! 5
Limitations of Modeling • Models are used to simulate rocket plume characteristics – Sensor testing, HWIL – Simulations – Algorithm testing • Pre-built databases of modeling results can alleviate the runtime limitations 6
Limitations of Modeling • Models are used to simulate rocket plume characteristics – Sensor testing, HWIL – Simulations – Algorithm testing • Pre-built databases of modeling results can alleviate the runtime limitations • Pre-built databases introduce other limitations: • Nearest neighbor image interpolations require large number of images and produces discontinuities in the extracted imagery • Inflexible to new conditions • Image registration techniques can be used to address these limitations 7
Outline • Introduction – Who is PST? – Why are we interested in image registration? • What is image registration? – A basic definition – Examples • Solution: Image Morphing Toolkit – The library – Examples • Conclusion 8
Definitions: Image registration: Process of aligning multiple images according to corresponding features in each. Image morphing: (an application image registration) Images are progressively aligned to common intermediate conditions, creating an interpolation between images. 9
Transformations Shear Translation Rotation Non-rigid transformation (warp) 10
Image Morphing • Image morphing is the calculation of transformed images that smoothly blend between two input images. • Corresponding features must be identified and tracked (using landmark points, in green below). • This requires a transformation of the two input images using the landmark points to an intermediate condition. • A simple average of the images would cause two overlapping ghosted images. 11
Program Motivation - Jitter • Tightly spaced database points minimize the step-wise changes between points • Jitter (discontinuities) still occur; especially when the image is changing quickly, or the viewing conditions change quickly • Morphing allows smooth interpolation of the images 12
Outline • Introduction – Who is PST? – Why are we interested in image registration? • What is image registration? – A basic definition – Examples • Solution: Image Morphing Toolkit – The library – Examples • Conclusion 13
Image Morphing Tool (IMT) • C++/CUDA library that fully automates image morphing procedures – Solves thin plate spline warping transformation – Incorporates hole filling methodology – Capable of working with multiple core computers as well as single CPU devices – GPU-enabled for real time morphing (500+ Hz) – Designed for general use • Application specific tools (e.g. PMT, for plumes) – Automated landmark identification tailored to missile plume characteristics provide feature landmark positions – Interface to source images from customer specified database 14
Thin-Plate-Spline Warps • The TPS algorithm defines a transformation f ( x , y ) ( X ( x , y ), Y ( x , y )) giving new coordinates for points in the original image. • The transformation is determined by minimizing the curvature of the surface (integrated squared second derivative) 2 2 2 2 I R ( f 2 f f ) x y f xx xy yy 15
IMT Flow Chart At high level, the process flow is relatively simple: • Given: set of input conditions • IMT routines to return landmarks and images. • IMT performs the required trans- formations, and returns the result. 16
Process Within the IMT Step 1 – Select images that will be morphed ‘from’ Step 2 – Retrieve landmarks on images Step 3 – Compute new landmarks Step 4 – Warp images to new landmarks Step 5 – Cross dissolve the warped images Landmarks from Landmarks from Image 1 Image 2 Solve for new landmarks Warp 1 Warp 2 Morph 17
Database Interpolation • Linear interpolation is currently used to determine target landmark positions • 4 input images are needed for interpolation in 2 dimensions. • In a database with 3 independent variables, 8 input images are needed • IMT determines the required images and retrieves them automatically 18
Code Example // Set up 3D morphing object getImageFcn imageDatabaseFcn(); getlmFcn landmarksfunc(); #ifdef CUDA morph3D_cuparam param(landmarksfunc.nlm(),ni,nj,3); typedef morph3D_cuda<getImageFcn,getlmFcn> morph3D; #elif PARALLEL TPSwarp_param param(landmarksfunc.nlm(),ni,nj); typedef morph3D_parallel<getImageFcn,getlmFcn> morph3D; #else // SERIAL TPSwarp_param param(landmarksfunc.nlm(),ni,nj); typedef morph3D_serial<getImageFcn,getlmFcn> morph3D; #endif float *imageMorph=new valueType[numPix]; morph3D mymorph3D(&nodes,&landmarksfunc,&imageDatabaseFcn,¶m); //Perform image interpolation (cond1, cond2, & cond3 must be defined) While (!done) { getConditions(*cond1,*cond2,*cond3); mymorph3D.morph(cond1,cond2,cond3,imageMorph); // do something with the image } delete[] imageMorph; 19
GPU Acceleration The IMT was developed to support • serial, • parallel (multi-core) and • CUDA • Windows and Linux. GPU accelleration: • Windows: ~60x’s increase in speed • Linux: ~80x’s increase in speed CUDA techniques: • Pinned memory transfers from CPU to GPU • cuBLAS functions • Custom kernels • CUDA Streams • Concurrent memory copies 20
GPU Acceleration First pass CUDA implementation: transform() initialize() transform() map() on intialize() on image 2, on image 1 on image 1 image 1 on image 2 etc Second pass CUDA implementation: initialize() all transform() map() all 8 8 images all 8 images images, etc Each piece of the IMT morph (initialize, transform, and map) are naturally parallel processes. Our first implementation focussed on the parallel nature of each process. In our second pass, however, we found that using “batched” methods (e.g. cublasSgetrfBatched(…) ), significantly accelerated the computation. 21
Performance Operating System Processor GPU Method Time in Hz Windows 7 i7-4820K Serial 9 Linux i7-4820K Serial 9 Windows 7 i7-4820K Parallel 37 Windows 7 i7-4820K Titan CUDA 530 Linux i7-4820K Titan CUDA 720 These frame rates were calculated for 128x256 pixel images, using 21 landmarks, interpolating in three database dimensions, operating on dual images. (Each “frame” here represents two images together.) 22
Additional Applications Thermal images of uniforms can be warped Coupled with registration techniques, allows a quantitative comparison Applications include: quantitative comparisons for thermal properties of textiles, degradation of parts and materials, part failure prediction. 23
Application Example Non-rigid image registration can be used to make quantitative comparisons between measurements of images that change with time. The images here show two MRI scans of a knee. The knee on the right is bent more than the knee of the left, making registration and differencing difficult. 24
Application Example Image comparison without registration Simply overlaying the images, and displaying the differences does not produce usable results. Rigid registration techniques, in this case, can do little better. 25
Application Example Image comparison with registration Non-rigid registration, as done with the IMT, can align features within the images, thus allowing for quantitative comparisons. 26
Future Applications • Apply measured data to increase fidelity of models through parametrically warping model predictions (increase realism) • Apply image morphing principles to 3D fields of data • Test IMT with new applications • Develop more tools to automate landmark selection 27
Recommend
More recommend
