Super-Resolution via Image Recapture and Bayesian Effect Modeling Neil Toronto Oral Thesis Defense Department of Computer Science Brigham Young University December 2008
Single-Frame Super-Resolution (i.e. Good Image Magnification) Printing photos from a camera or the Internet Compositing images Signal conversion (e.g. DVD to HDTV) Crime drama television, blackmail Generally an underconstrained, ill-posed problem General solution: make it well-posed and make assumptions, then infer the extra information Super-Resolution via Image Recapture and Bayesian Effect Modeling 2
Size vs. Apparent Resolution Bilinear Nearest neighbor (NN) Input Sinc ??? Super-Resolution via Image Recapture and Bayesian Effect Modeling 3
Nonadaptive Methods Assume the pixels are a function or signal sample Families: function-fitting, frequency-domain Implementation of both: sum up scaled copies of the same fuzzy shape (kernel) Artifacts: Blocky: Blurry: Bilinear Bicubic Sinc Only two possible solutions: get more data, make stronger assumptions Super-Resolution via Image Recapture and Bayesian Effect Modeling 4
Adaptive Methods Make strong assumptions Families: edge-preserving, training- based, optimization Examples: Resolution synthesis (RS), local correlation (LCSR): learn optimal kernels Local correlation (LCSR) from examples, apply locally according to class Image analogies, Freeman’s MRFs: Construct Frankenimages from Flickr Level-set reconstruction: optimize upscaled result with respect to rewarding accuracy and penalizing jaggies Resolution synthesis (RS) Super-Resolution via Image Recapture and Bayesian Effect Modeling 5
Quantitative Comparison Popularly mean-squared error on downscaled and Original images reconstructed images, results compared against nonadaptive whipping-boys Decimation Ouwerkerk 2006: First ever qualitative and quantitative survey of super-resolution Super- resolution Tested nine Correctness methods on ? measures ? seven test ? images using three measures of correctness The winners: resolution synthesis (RS) and local correlation (LCSR) Reconstructed images Super-Resolution via Image Recapture and Bayesian Effect Modeling 6
Motivation 2x 4x LCSR RS Bayesian edge inference (BEI) Objective: avoid these artifacts, be competitive on Ouwerkerk’s measures Super-Resolution via Image Recapture and Bayesian Effect Modeling 7
Optimization Reconstruction Framework Reconstruction the mostly Bayesian way Assumptions: an image I ’ existed that was degraded to produce I Image prior Task: given I , reconstruct I ’ Reconstruct using Bayes’ Law: Degradation P I' ∣ I = P I ∣ I' P I' P I Result is almost always argmax P ( I ’| I) (a “MAP estimate”) Super-Resolution via Image Recapture and Bayesian Effect Modeling 8
Modeling Mismatch 1: Printing a Full-Resolution Photo There’s no higher-resolution original image to reconstruct Super-Resolution via Image Recapture and Bayesian Effect Modeling 9
Modeling Mismatch 2: CCD Demosaicing There’s no pristine, unfiltered original image to reconstruct Super-Resolution via Image Recapture and Bayesian Effect Modeling 10
Recapture Reconstruction Framework Assumptions: a scene S was captured with C to create image I Task: given I (and possibly C ), reconstruct S , and recapture it as I ’ using a fictional C ’ P I' ∣ I =⋯ Super-Resolution via Image Recapture and Bayesian Effect Modeling 11
Using the Recapture Framework 1 Define scene model S 1 1 1 2 Define capture process I | S , C 2 and recapture process I ’| S , C ’ 3 Sample I ’| I (the “posterior 3 3 3 2 2 predictive” distribution) 3 ’s inference is familiar, but not always easy 3 Super-Resolution via Image Recapture and Bayesian Effect Modeling 12
Modeling Step Edges Goal: preserve edges and gradients Assumption: scenes are mostly comprised of solid objects with coherent boundaries Capture ≈ global blur + sampling at discrete points Discrete sampling Point-spread Scene model: a grid of linear discontinuities convolved with blurring kernels (appx. spatially varying PSF) I 00 I 01 I 02 = I 10 I 11 I 12 * Discrete I 20 I 21 I 22 sampling S Super-Resolution via Image Recapture and Bayesian Effect Modeling 13
Spatially Varying Point-Spread Spatially varying point spread Dark = narrow BEI’s reconstruction 2x magnification Super-Resolution via Image Recapture and Bayesian Effect Modeling 14
Scene Model: Causal or Noncausal? Hierarchical Compatibility (causal) (noncausal) Super-Resolution via Image Recapture and Bayesian Effect Modeling 15
Utility of Compatibility No compatibility Compatibility Samples from prior predictive distribution I ’ No data Samples from posterior predictive distribution I ’| I Data 4x magnification Super-Resolution via Image Recapture and Bayesian Effect Modeling 16
Bayesian Effect Modeling Compatibility is an effect (noncausal) Capture is obviously causal Need to mix the two in the same model Solution: use the transformation from MRFs to BNs, but locally Confines noncausal dependence to local subgraphs Can’t create cycles Super-Resolution via Image Recapture and Bayesian Effect Modeling 17
Minimum Blur and Decimation Blur Doing super-resolution without accounting for point- spread gives blurry results I ’ is naturally sharpened by adding minimum blur variance in capture model and adding less in recapture Decimation: minimum blur converging on σ = 1/3 ¼ ¼ ¼ ¼ * ••• * * * ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ Total σ 2 = 1/9 If decimation has occurred, model it by setting minimum blur to 1/3 in capture and 1/(3 s ) in recapture Super-Resolution via Image Recapture and Bayesian Effect Modeling 18
Results: Boundary Coherence Decimated (NN) BEI RS LCSR 4x magnification Super-Resolution via Image Recapture and Bayesian Effect Modeling 19
Results: Edges and Gradients Original Decimated (NN) Bilinear NEDI LCSR RS BEI BEI 8x 4x magnification after two decimations Super-Resolution via Image Recapture and Bayesian Effect Modeling 20
Results: Correctness Measures Super-Resolution via Image Recapture and Bayesian Effect Modeling 21
Complementary Breakfast: CCD Demosaicing Missing data problem? Rev. Bayes says it’s easy: just leave it out In CCD demosaicing tasks, 2/3 of the data is missing: it was never collected Original Bicubic interpolation BEI’s reconstruction Super-Resolution via Image Recapture and Bayesian Effect Modeling 22
Complementary Breakfast: CCD Demosaicing Original Bicubic interpolation BEI’s reconstruction Super-Resolution via Image Recapture and Bayesian Effect Modeling 23
Complementary Breakfast: Inpainting and Restoration Inpainting can be seen as a missing-data problem Defaced 33% BEI’s reconstruction Could model defacement in the capture process Super-Resolution via Image Recapture and Bayesian Effect Modeling 24
Complementary Breakfast: Inpainting and Restoration Defaced 33% BEI’s reconstruction Super-Resolution via Image Recapture and Bayesian Effect Modeling 25
Limitations and Future Work Slow: could just be Python’s fault, but there’s a lot to compute Super-resolution’s canny ridge Line and T-junction models Throwing the recapture framework at every reconstruction problem that still twitches Making Bayesian effect modeling into a graphical model in its own right Super-Resolution via Image Recapture and Bayesian Effect Modeling 26
Take-Home Messages In super-resolution (or reconstruction), make strong assumptions cuz you ain’t gettin mo data Model capture, reconstruct scenes, recapture results Potentially fits the actual process better More flexible than modeling just degradation Explicitly model what you want to reason about precisely (e.g. edges, sharpness, scenes) Compatibility is more tractible than hierarchy and gives great results Missing data is no problem for Bayesians Super-Resolution via Image Recapture and Bayesian Effect Modeling 27
Questions ? ? Super-Resolution via Image Recapture and Bayesian Effect Modeling 28
An Intriguing Equivalence Reconstruction Supervised via recapture machine learning ≡ Super-Resolution via Image Recapture and Bayesian Effect Modeling 29
Details: Effect Modeling Compatibility to conditional density conversion: Joint density: Unnormalized complete conditional (Markov blanket) density: Super-Resolution via Image Recapture and Bayesian Effect Modeling 30
Details: Definitions Given an m x n image I . Image coordinates are properly a parameter of the capture process C . C ’ also contains an array of coordinates. Compatibility and capture are defined in terms of nearest neighbors: Super-Resolution via Image Recapture and Bayesian Effect Modeling 31
Details: Step Edges Step edge geometry is expressed as an implicit line: Facet profiles are defined by Gaussian convolution and have an analytic solution: Because of symmetry, 2D step edges can be expressed in terms of their profiles . Super-Resolution via Image Recapture and Bayesian Effect Modeling 32
Recommend
More recommend
