Obs. Strasbourg - Apr 13, 2007 PASEO DeepSkyFusion* multisource data fusion from astronomical images André Jalobeanu PASEO Research Group MIV team @ LSIIT, Illkirch, France *part of SpaceFusion project ANR “Jeunes Chercheurs” 2005-2008
Outline Introduction Objectives (Astronomy) Proposed approach Bayesian inference from multiple observations Accurate forward modeling Preliminary results: validation in 1D and 2D Estimating, storing and using uncertainties Extensions to multispectral/hyperspectral data Multispectral imaging Integral Field Spectroscopy (Muse)
Why multisource data fusion? Problem: lots of data, same object! Images are recorded with various: � position, orientation (pose) � sensors (resolution, noise, bad pixels) � observing conditions (transparency, haze) � instruments (blur, distortions) � Multisource data fusion: � Optimally combine all observations into a single object � Preserve all the information from the original data set • Increase resolution if needed • Compute the uncertainties • Reconstruct the 3D geometry if required � Enhance the object quality (optional) Denoise or deblur depending on the degradation
The SpaceFusion project Projet ANR “Jeunes Chercheurs 2005” (French Research Agency) 3-year grant, Jan 2006 - Dec 2008 Name Position, lab time% André Jalobeanu CR, LSIIT/MIV Illkirch 90% Christophe Collet PU, LSIIT/MIV Illkirch 40% Mireille Louys MCF, LSIIT/MIV Illkirch 40% Fabien Salzenstein MCF, InESS Strasbourg 40% Françoise Nerry CR, LSIIT/TRIO Illkirch 20% Albert Bijaoui / Eric Slezak A/AA, OCA Nice 10% Bernd Vollmer A, Obs. Strasbourg 10% J.A. Gutiérrez (06) postdoc, Illkirch 20 mo total F. Brisc, M. Joshi (07)
SpaceFusion objectives Produce a corrected, super- resolved image in astronomy Reconstruct a reflectance function in remote sensing Recover the geometry of small bodies and planetary surfaces Reconstruct both reflectance and topography in Earth/Space Sciences
Astronomy: image reconstruction DeepSkyFusion Multisource data fusion and 2D super-resolution Astronomy & Astrophysics � Input: � Multiple images (single band, multispectral or IFS) Virtual Observatory � Optical, UV, IR / calibrated or not / missing or corrupted data � Output: � Single model: 2D (image-like), well-sampled � Uncertainties (simplified inverse covariance) � If applicable, spatial and spectral super-resolution Preserve the information from the original data set: photometry, astrometry, noise statistics
DeepSkyFusion project: task layout Astronomy, 2D - DeepSkyFusion independent Dynamic range fusion 0D pixel processing Dynamic range fusion with PSF Intensity/uncertainty quantization Missing data and outlier management Noise modeling, handling, estimation Multispectral data fusion 2D image Rendering (2D image formation) PSF inference processing Efficient image modeling 2D Image fusion, single, known param. Format specification Image fusion, multi, known param. DeepSkyFusion Registration Oct. 2006 Efficient optimization schemes status Parameter estimation via marginalization Theory defined task Covariance simplification partly completed task Recursive inference completed and validated task Error propagation (processing chains)
Related, existing methods (2D) � Frame coaddition, Drizzling � Increase the SNR and the dynamic range � Mosaicing, “Montage” � Increase the field of view � Super-resolution � De-aliasing � Bandwidth extrapolation � Multiframe restoration � Partial turbulence compensation
The proposed approach (single band images) resolution camera geometric � pose mapping parameters observed Use Bayesian inference to recover a rendering image f coefficients Y � single object from all observations internal � camera parameters h � , In 2D: recover a well-sampled � global � PSF noise sensor sampling grid image, possibly super-resolved Check the validity of this approach in 1D & 2D ( first results ) Provide uncertainty estimates, allow for recursive data processing
Bayesian inference prior model OBJECTIVE : likelihood (a priori knowledge posterior probability image formation model about the observed object) density function (pdf) p ( θ | observations ) = p ( observations | θ ) × p ( θ ) p ( observations ) parameters of interest (unknown solution) evidence (useful for model comparison) ! All parameters are random variables ! Bayesian inference � functional optimization / approximations ! Deterministic optimization techniques for speed
Probabilistic data fusion vs. averaging Probabilistic fusion Average Result #2 Result #1 � Take into account uncertainties: variance, correlations Formal framework for the combination of multiple observations � Propagate uncertainties From the observation noise to the end result! Downside: algorithms ought to account for input uncertainties
Bayesian inference from multiple observations Y � image model Y � camera X � parameters parameters Y � unknown object observations � Forward modeling: • Object modeling (image, 3D geometry, reflectance...) • Image formation = rendering + degradations � Bayesian inference: • Estimate the optimal object given all observations • Integrate w.r.t. all nuisance variables: “marginalization” • Evaluate the uncertainties: • Model selection and assessment Covariance matrix, if Gaussian approx. of the posterior pdf
Band-limited astronomical image model � Model of the unknown object ( 2D image ) � Choose an appropriate parametrization and topology • Sampling grid size � , lattice, ... 2 / e z i s r u b l t u p n i < � Understand the sampling theorem! e z s i e l x p i t u p t • Don’t try to go beyond the Nyquist rate : u O • Near-optimal sampling, band-limited: optical frequency cut-off! Target image = sum of B-Spline 3 kernels � Constrain and stabilize this inverse problem • Use smoothness priors to avoid noise amplification • Use efficiently designed prior models (oversampled areas will undergo a deconvolution...) to help preserve useful information while filtering the noise
Simplified graphical model prior model scene model � � marginalization parameters resolution scene X camera pose Y � model parameters observed image n observations Directed graphical models: � Random variable Node: set of random variables No incoming arrow: prior pdf Y Observed variable Arrow: dependence (causality) Set of incoming arrows: conditional pdf Joint distribution: P(X,Y, � , � , � ) = P( � ) P( � ) P(X| � ) � n P( � n ) P(Y n |X, � n , � ) Posterior marginal: P(X|Y) ∝ � P(X,Y, � , � , � ) d � d � d �
Resampling scheme Image formation scheme [MaxEnt 2006] � Deformation Geometric mapping g (param: external � , internal � , density � ) � W θ g s � g ( u ) = ǫ α ( u )+ b θ � Convolution with the Point Spread Function (PSF) h � Sampling on a discrete pixel grid � I p = ( T ◦ g ) ⋆ h ( π p ) For each sensor n: � I n λ n p = pj L j L j : BSpline interpolation j coefficients such that where � = function of g, h target image X=L ! s
Rendering coefficient computation Sensor space Model (world) space [ADA 2006] g Instrument PSF (at pixel p) p Sampling grid � (irregular) n instruments Warped, shifted PSF j Target PSF (B-Spline 3) Rendering coefficients: � � � �� � h n � � λ n � ∆ n ∆ n � pj = g ( π p ) − j p p p Jacobian of the p = J − 1 notation ∆ n geometric mapping g π p n th sensor PSF @ pixel p ation at pixel p no
Inversion (known parameters) Energy U = - log P(L | Y, � , � , � ) 1) Optimum: functional minimization problem 2) Uncertainties: 2nd derivatives at the optimum � U ( L ) = Φ ( L ) + D n ( L, Y n ) n Φ ( L ) = ω � DX � 2 = ω � RL � 2 Prior term : R = DS = d ⋆ s ⋆ BSpline derivative kernel (1st order) 2 � � D n ( L, Y n ) = 1 λ n p · L − Y n � Data term : p 2 v n p p
Inversion algorithm � Initialization (... looks like drizzling!) n,p ( λ n p ) j Y n � p L j ← � n,p ( λ n p ) j � Iterative, deterministic optimization Quadratic form: conjugate gradient minimization of U(L) � λ n p · L − Y n � ∇ U = 2 ω R T RL + p � Gradient λ n p v n p n,p 1 [ ∇ 2 U ] = 2 ω R T R + � ( λ n p ) ( λ n p ) T Hessian v n p n,p
First results: 2X super-resolution (1D signals) [MaxEnt 2006] (model space) ideal signal data points 2 observations blur, noise 1/3 sample shift real PSF Reconstructed signal De-aliasing 95% confidence interval -1 0 1 + regularized deconvolution target PSF
2D Super-Resolution results (astronomy) 3 2 0 1 0 1 pointing pattern (model space) Image fusion result (mean) 2 3 Experimental setting: 4 noisy images undersampled (factor 2), shifted (1/2 pixel) [ADA 2006] Inverse covariance of the result diagonal terms, and near-diagonal (covariance) terms
Comparison with existing techniques Reference image Initialization Image fusion result (mean) Pixel interlacing result Drizzling result
Real data? (HST WFPC2) DITHER HANDBOOK - Example #2: EDGE-ON GALAXY NUCLEUS NGC 4565 bright dots: mostly cosmics 4 images of the same scene, shifted, undersampled
Recommend
More recommend