Real-Time Non-Rigid Image Registration (Image Morphing) Engine - - PowerPoint PPT Presentation

Real-Time Non-Rigid Image Registration (Image Morphing) Engine

Propulsion Science and Technology, Inc. (PST) Randall Miles, Donna Yosmanovich, Michael Thorwart, Margaret Taylor GPU Technology Conference 2016 San Jose, CA April 5, 2016 1

PST TP - 215

Willow

2

https://goo.gl/0OxGAk

The film, Willow, directed by George Lucas and staring Warwick Davis, was the first example of extensive use of image morphing in film. Dennis Muren, Creative Director at Industrial Light and Magic, said the scene took weeks of computer work at the time, but could be done on a cell phone today.

Example: Image Morphing

3

Michael Jackson’s music video “Black and White” is another early example of image morphing. In the video, faces of individuals of various nationalities are morphed from one to another as they dance and smile on the screen. The frame shown here capture the transition from a western male to an asian female.

Outline

4

• Introduction

– Who is PST? – Why are we interested in image registration?

• What is image registration?

– A basic definition – Examples

• Solution: Image Morphing Toolkit

– The library – Examples

• Conclusion

Who is PST?

(Propulsion Science and Technolgy)

5

Scientists with a passion for

• Science
• Technology
• and rockets!

Our historical focus: modeling exhaust plumes and wakes Physics based modeling and simulation is time consuming!

Limitations of Modeling

• Models are used to simulate rocket plume characteristics

– Sensor testing, HWIL – Simulations – Algorithm testing

• Pre-built databases of modeling results can alleviate the

runtime limitations

6

Limitations of Modeling

• Models are used to simulate rocket plume characteristics

– Sensor testing, HWIL – Simulations – Algorithm testing

• Pre-built databases of modeling results can alleviate the

runtime limitations

7

• Pre-built databases introduce other limitations:
• Nearest neighbor image interpolations require large number of images

and produces discontinuities in the extracted imagery

• Inflexible to new conditions
• Image registration techniques can be used to address these

limitations

Outline

8

• Introduction

– Who is PST? – Why are we interested in image registration?

• What is image registration?

– A basic definition – Examples

• Solution: Image Morphing Toolkit

– The library – Examples

• Conclusion

Definitions:

Image registration: Process of aligning multiple images according to corresponding features in each. Image morphing: (an application image registration) Images are progressively aligned to common intermediate conditions, creating an interpolation between images.

9

Transformations

10

Translation Rotation Shear Non-rigid transformation (warp)

Image Morphing

11

• Image morphing is the calculation of transformed images that smoothly

blend between two input images.

• Corresponding features must be identified and tracked (using landmark

points, in green below).

• This requires a transformation of the two input images using the

landmark points to an intermediate condition.

• A simple average of the images would cause two overlapping ghosted

images.

Program Motivation - Jitter

• Tightly spaced database points

minimize the step-wise changes between points

• Jitter (discontinuities) still occur;

especially when the image is changing quickly, or the viewing conditions change quickly

• Morphing allows smooth interpolation of

the images

12

Outline

13

• Introduction

– Who is PST? – Why are we interested in image registration?

• What is image registration?

– A basic definition – Examples

• Solution: Image Morphing Toolkit

– The library – Examples

• Conclusion

Image Morphing Tool (IMT)

• C++/CUDA library that fully automates image

morphing procedures

– Solves thin plate spline warping transformation – Incorporates hole filling methodology – Capable of working with multiple core computers as well as single CPU devices – GPU-enabled for real time morphing (500+ Hz) – Designed for general use

• Application specific tools (e.g. PMT, for plumes)

– Automated landmark identification tailored to missile plume characteristics  provide feature landmark positions – Interface to source images from customer specified database

14

Thin-Plate-Spline Warps

• The TPS algorithm defines a transformation

giving new coordinates for points in the

• riginal image.
• The transformation is determined by

minimizing the curvature of the surface (integrated squared second derivative)

15

)) , ( ), , ( ( ) , ( y x Y y x X y x f 



     y x ) f f f ( R I

yy xy xx f 2 2 2 2

2

IMT Flow Chart

16

At high level, the process flow is relatively simple:

• Given: set of input

conditions

• IMT routines to return

landmarks and images.

• IMT performs the

required trans- formations, and returns the result.

Process Within the IMT

Landmarks from Image 1 Landmarks from Image 2 Warp 1 Warp 2 Morph Solve for new landmarks

17

Step 1 – Select images that will be morphed ‘from’ Step 2 – Retrieve landmarks on images Step 3 – Compute new landmarks Step 4 – Warp images to new landmarks Step 5 – Cross dissolve the warped images

Database Interpolation

18

• Linear interpolation is

currently used to determine target landmark positions

• 4 input images are

needed for interpolation in 2 dimensions.

• In a database with 3

independent variables, 8 input images are needed

• IMT determines the

required images and retrieves them automatically

Code Example

// Set up 3D morphing object getImageFcn imageDatabaseFcn(); getlmFcn landmarksfunc(); #ifdef CUDA morph3D_cuparam param(landmarksfunc.nlm(),ni,nj,3); typedef morph3D_cuda<getImageFcn,getlmFcn> morph3D; #elif PARALLEL TPSwarp_param param(landmarksfunc.nlm(),ni,nj); typedef morph3D_parallel<getImageFcn,getlmFcn> morph3D; #else // SERIAL TPSwarp_param param(landmarksfunc.nlm(),ni,nj); typedef morph3D_serial<getImageFcn,getlmFcn> morph3D; #endif float *imageMorph=new valueType[numPix]; morph3D mymorph3D(&nodes,&landmarksfunc,&imageDatabaseFcn,&param); //Perform image interpolation (cond1, cond2, & cond3 must be defined) While (!done) { getConditions(*cond1,*cond2,*cond3); mymorph3D.morph(cond1,cond2,cond3,imageMorph); // do something with the image } delete[] imageMorph;

19

The IMT was developed to support

• serial,
• parallel (multi-core) and
• CUDA
• Windows and Linux.

GPU accelleration:

• Windows: ~60x’s increase in speed
• Linux: ~80x’s increase in speed

CUDA techniques:

• Pinned memory transfers from CPU to GPU
• cuBLAS functions
• Custom kernels
• CUDA Streams
• Concurrent memory copies

GPU Acceleration

20

GPU Acceleration

21

initialize()

• n image 1

transform()

• n image 1

map() on image 1 intialize()

• n image 2

transform()

• n image 2,

etc

initialize() all 8 images transform() all 8 images map() all 8 images, etc

First pass CUDA implementation: Second pass CUDA implementation: Each piece of the IMT morph (initialize, transform, and map) are naturally parallel processes. Our first implementation focussed on the parallel nature of each process. In our second pass, however, we found that using “batched” methods (e.g. cublasSgetrfBatched(…)), significantly accelerated the computation.

22

Performance

Operating System Processor GPU Method Time in Hz Windows 7 i7-4820K Serial

9

Linux i7-4820K Serial

9

Windows 7 i7-4820K Parallel

37

Windows 7 i7-4820K Titan CUDA

530

Linux i7-4820K Titan CUDA

720

These frame rates were calculated for 128x256 pixel images, using 21 landmarks, interpolating in three database dimensions, operating on dual images. (Each “frame” here represents two images together.)

Additional Applications

23

 Thermal images of uniforms can be warped  Coupled with registration techniques, allows a quantitative comparison  Applications include: quantitative comparisons for thermal properties of textiles, degradation

• f parts and materials, part failure

prediction.

Application Example

24

Non-rigid image registration can be used to make quantitative comparisons between measurements of images that change with time. The images here show two MRI scans of a knee. The knee on the right is bent more than the knee of the left, making registration and differencing difficult.

Image comparison without registration

Application Example

25

Simply overlaying the images, and displaying the differences does not produce usable results. Rigid registration techniques, in this case, can do little better.

Image comparison with registration

Application Example

26

Non-rigid registration, as done with the IMT, can align features within the images, thus allowing for quantitative comparisons.

Future Applications

• Apply measured data to increase fidelity
• f models through parametrically

warping model predictions (increase realism)

• Apply image morphing principles to 3D

fields of data

• Test IMT with new applications
• Develop more tools to automate

landmark selection

27

IMT Solution

• Jitter

– IMT resolves jitter issues – Allows smoothly varying interpolations over any series

• f conditions
• Reduce Database Size and Cost

– Based on current error analysis, implementing the IMT can reduce database size by an order of magnitude

• Quantitative comparisons for data analytics
• Image stabilization for deforming images
• Extreme high-speeds possible due to GPU

acceleration

• This capability could be deployed in embedded

systems, on boards like the Jetson

28

Acknowledgements

I’d like to acknowledge the Air Force SBIR program for their investment, and thank Nvidia for their generous support of our program.

29