Parallel Algorithms and Optimization for Multi-Aperture Image - PowerPoint PPT Presentation

Parallel Algorithms and Optimization for Multi-Aperture Image Superresolution Reconstruction Bob Plemmons with T. Guy and P. Zhang Wake Forest and the Project Research Team Sponsored by ARO/DTO/IARPA 1 1 1

Outline  Introduction: Comments on Superresolution (SR)  Fusing arrays of images  PERIODIC Research Project  Inverse Problems Considered  Block Toeplitz Matrix Structures & Parallel SR Algorithms  Application-specific integrated circuit platforms  FPGA implementation  Test Results 2

INTRODUCTION - Image Resolution Image resolution describes the detail an image holds. The following figure provides how the same image appears at different pixel resolutions. Example without noise or blur: Superresolution (SR) – digital means for increasing spatial resolution • Low resolution images should be different “looks” of the same scene. • Our interest – HR image for arrays of LR images taken using a single detector camera – snapshot image. 3

Multiple Aperture Imaging  Limited utility in the analog age of cameras since photochemistry provides no means for fusing images  Insure against failure of a single aperture  Increases the field of view (about 180º x 120º for us)  Allows acquisition of multispectral, polarization, and multiscale data  Emerging computational systems lead to performance advantages  Multiple apertures common in living systems (including us) 4

Compound Lens of Dragonfly: A Multiple Aperture System

History: TOMBO: Multi-aperture Camera “Thin observation module by bound optics(TOMBO),” J. Tanida, et al., Applied Optics, 2001

(Practical Enhanced Resolution Integrated Optical Digital Imaging Cameras) PERIODIC Project 2005 – 2009 7

Our Compound Lens Camera for Biomedical Studies • A multi-spectral 18 lens imager • 18 narrow line-width filters (400-800 nm) • Integrated illumination • Single large format 10.6 Mpixel CCD • Used at: WFU Institute for Regenerative Medicine Printing Skin for Grafting

3 x 3 Array Prototype Camera 3 x 3 LOW-DEF images post- processed into one or more HIGH-DEF images 9

Superresolution Computations Deblurring • l = up-sampling factor • Low resolution images should be different “looks” of the same scene. • They should have different sub pixel shifts. 10

Image Registration Image Registration is the process of spatially matching two images, i.e., the reference and target images, so that corresponding coordinate points in the two images correspond to the same physical region of the scene being imaged. Followed by interpolation (splines, etc) onto high resolution grid. 11

INVERSE PROBLEM CONSIDERED 12

Superresolution/Deblurring – Forward Model Image Formation Model: A low-resolution image, g j formed by the j th lenslet is given by : g j = D H j S j f + η j  f is a target high-resolution representation for object,  S j represents the translation of image f due to the relative position of lenslet j with respect to the object and the detector (shift)  H j is a blurring operator associated with lenslet j,  D is a decimation matrix,  g j is the low-resolution image associated with lenslet j,  η j describes noise process associated with j th image.  Compact notation: g = Af + η 13

Inverse Problem  Produce a high-resolution image from low-resolution lenslet images. g A 2     g DH S 1 1 1     g DH S     = − + λ 2 2 2 J ( f ) f R ( f )               g DH S m m m 2 m = l 2 , where l is the up-sampling factor.  The blurring, decimation, and shift matrices need to be determined. Various types of penalty (regularization) terms can be used. Speed is a important in our case (FPGA implementation) . = ˆ argmin { ( )} f J f f 14

Superresolution Reconstruction Example Up-sampling factor 4. Sixteen LR images taken using PERIODIC camera. The sixteen 128 x 128 images used to reconstruct a 512 x 512 HR image. l = 4. Reconstructed HR LR Subimage 15 15 15

Computation Bottleneck of SR  Concatenate the product matrices into matrix A. = − 2 + λ 2  Inverse problem: min ( ) J f Af g f 2 2 • Large matrix computation: given sixteen 128 x 128 LR images, we construct a 512 x 512 HR image. Normally, matrix A would be of a size 262,144 x 262,144 . We seek a sparse solution method.  The original matrix A has a two-level tri-diagonal block Toeplitz structure, and can be reduced to 9 smaller matrices, of size l 2 x l 2 , where l is the up-sampling factor.  We use nine 16 x 16 matrices for the example above.  Key step is rearranging the arrays based on properties of D , H , S A . and produce a structured matrix i i 16 16 16 16

Structures of Superresolution Matrices Without blurring, A = DS. For the array imaging SR problem there exists permutation matrices P and Q such Q T That has the two-level block Toeplitz structure: AP 17 17 17

Structures of Superresolution Matrices With blurring, A = DHS. P T A T AP has the same structure as Q T AP below, two-level block Toeplitz. 18 18 18

Designing Algorithms by Exploiting Matrix Structures Without blurring, A = DS. Notation : l = l After even-odd (red-black) permutations in both rows and columns, matrix A T Af = A T g becomes Solve with Combination of Block Gauss-Siedel (BGS) and Cyclic Reduction (CR). 19 19 19 19

Designing Algorithms by Exploiting Matrix Structures With blurring, A = DHS, with spread of H no more than 2 l +1 pixels. Again, solve with Block Gauss-Siedel method. 20 20 20

Features of the Algorithms  Parallelism: At each step, we solve n/2 l much smaller scale sub- problems that can be solved independently by many processors. This greatly enhances the throughput of the system.  Small scale computations: The main operations involved are matrix-vector multiplications on the scale of l 2 .  FPGA-implementation: With these algorithms, the SR problem is now practical to solve on parallel systems like FPGAs. 21 21 21

Application-specific Hardware Implementation Curt Petersen, EM Photonics Inc. 22

Array Based Digital Super Resolution Hardware Implementation – High is Best Estimated Affordability Power Flexibility Performance (1/cost) (FLOPS/watt) GPUs Very High Low Very Low Very High (**GPU development systems for embedded applications are not yet available) FPGAs High Medium High High DSP High Medium Medium High Multicore Medium High Low Very High MPs CELL High Medium Very Low Medium 23 23

Array Based Digital Superresolution Hardware Implementation Features FPGA Development Platform • Virtex 5 SX50T FPGA • 52K Logic Cells • 594KB BlockRAM • 288 DSPs • 480 IO • 256MB DDR2 SDRAM • DVI Output • Gigabit Ethernet • Serial Terminal Debug • USB JTAG Programming 24 24

Digital Super Resolution using FPGA: System Architecture Internal FPGA Super Resolution Packaging: Memory Kernel External SDRAM • Protection from the elements and ruggedization. Memory • Redesign for desired form factor. Power: • Battery integration for portable use. Internal FPGA System Control Memory (RISC Soft Processor) Input/Output 25

Digital Superresolution using Embedded Hardware: Processing Steps (1) Image Capture • Raw 10.6 Mpixel image (2) Image Segmentation and Registration • Segment and register raw image into 16 low resolution (128x128) subimages • Registration at sub-pixel resolution (3) Digital Super Resolution Reconstruction • Reconstruct 512x512 high resolution image • Executed by FPGA • Primarily matrix-vector multiplications 26

TEST RESULTS 27

Superresolution Reconstruction A 10.6 MPixel raw image from the PERIODIC camera. We select sixteen 128 x 128 subimages to be used during reconstruction. 28 28 28

Sample Test Result A set of 16 LR images taken using the PERIODIC camera. The sixteen 128 x 128 images were used to reconstruct a 512 x 512 HR image. Speedup factor 12. LR Subimage Resolution Reconstructed HR enhancement ~ 2.0X note: after registration only 10 non-redundant subimages 29 29 29

Sample Test Result A set of 16 LR images taken using the PERIODIC camera. The sixteen 128 x 128 images were used to reconstruct a 512 x 512 HR image. (Walter Peyton) Reconstructed HR LR Subimage 30 30 30

Sample Test Result Reconstructed HR Reconstructed HR LR Subimage registration centered registration centered on object in front on object in back • Subpixel registration is sensitive to object distance . Whole scene digital superresolution would require a spatially varying registration algorithm. • Since depth information is embedded in the multiple subimages, this approach can be exploited to develop a sensitive and inexpensive imaging system that captures 3D information (depth) and performs digital superresolution simultaneously . 31 31 31

Summary  Superresolution (SR) – compound lens systems  PERIODIC Research Project  Inverse Problem for array imaging SR  Block Toeplitz Matrix Structures & Parallel SR Algorithms  Application-specific FPGA implementation  Test Results – Speedup factor about 12 vs implementation on 3.1Ghz 32 bit laptop 32

33