Sampling Resampling Warping Morphing Dr. Shai Avidan Faculty of - PowerPoint PPT Presentation

Sampling Resampling Warping Morphing Dr. Shai Avidan Faculty of Engineering Tel-Aviv University

Slide Credits (Partial List) • Alyosha Efros • Steve Seitz • Rick Szeliski • Bill Freeman • Fredo Durand

Sampling/Resampling/Warping/Morphing/Retargeting • Sampling – Given a continuous signal g(x) and its sampled counterpart g_s(x), are the samples of g_s(x) sufficient to exactly describe g(x)? – If so, how can g(x) be reconstructed from g_s(x)? • Resampling – Image resampling is the process of transforming a sampled image from one coordinate system to another • Warping – Is a geometric transformation that redefines the spatial relationship between points in the image • Morphing – Is the process of blending two images • Retargeting – Is a content-aware resizing operation

Sampled representations • How to store and compute with continuous functions? • Common scheme for representation: samples – write down the function’s values at many points [FvDFH fig.14.14b / Wolberg]

Reconstruction • Making samples back into a continuous function – for output (need realizable method) – for analysis or processing (need mathematical method) – amounts to “guessing” what the function did in between [FvDFH fig.14.14b / Wolberg]

1D Example: Audio low high frequencies

Sampling in digital audio • Recording: sound to analog to samples to disc • Playback: disc to samples to analog to sound again – how can we be sure we are filling in the gaps correctly?

Sampling and Reconstruction • Simple example: a sine wave

Undersampling • What if we “missed” things between the samples? • Simple example: undersampling a sine wave – unsurprising result: information is lost

Undersampling • What if we “missed” things between the samples? • Simple example: undersampling a sine wave – unsurprising result: information is lost – surprising result: indistinguishable from lower frequency

Undersampling • What if we “missed” things between the samples? • Simple example: undersampling a sine wave – unsurprising result: information is lost – surprising result: indistinguishable from lower frequency – also was always indistinguishable from higher frequencies – aliasing : signals “traveling in disguise” as other frequencies

Aliasing in video Slide by Steve Seitz

Aliasing in images

What’s happening? Input signal: Plot as image: x = 0:.05:5; imagesc(sin((2.^x).*x)) Alias! Not enough samples

Antialiasing • What can we do about aliasing? • Sample more often – Join the Mega-Pixel craze of the photo industry – But this can’t go on forever • Make the signal less “wiggly” – Get rid of some high frequencies – Will loose information – But it’s better than aliasing

Preventing aliasing • Introduce lowpass filters: – remove high frequencies leaving only safe, low frequencies – choose lowest frequency in reconstruction (disambiguate)

Sampling and Reconstruction

Image Acquisition Imaging Sampling g ( x,y ) g s ( x,y ) g d ( x,y ) f ( x,y ) subsystem Subsytem Quantizer * h ( x,y ) * s ( x,y ) Scene Image Sampled Digitized Image Image The scene is mapped to the image via the g ( x,y ) =f ( x,y ) *h ( x,y ) point spread function of the camera h ( x,y ) The continuous image g ( x,y ) then enters g s ( x,y ) =g ( x,y ) s ( x,y ) A sampling subsystem 2D comb function � � s ( x,y ) given by: � � � � � � � � � � s x , y x m , y n � �� � �� m n Therefore, g s (x,y) is now a discrete-continuous image with intensity values defined only over integral indices of x and y After sampling, the quantizer takes the continuous Intensity values and quantizes them (either uniform or non-uniform)

Sampling (I) We start with continuous signal f(x). Due to the psf We obtain a degraded, band-limited output g(x). The frequency content of g(x) is given by its spectrum, G(f), as determined by the Fourier transform: � � � � � � � � � i 2 fx G f g x e dx � � � � � � � � � � � � i 2 fx e cos 2 fx ic sin 2 fx Where x represents the spatial position and f denotes Spatial frequency. Since g(x) does not have any frequencies beyond f max , we say that it is band-limited. |G(f)| -f max f max

Sampling (II) The continuous output g(x) is then digitized by an ideal Impulse sampler, the comb function, to get the sampled signal g s (x). The ideal 1-D sampler is given by: � � � � � � � � � s x x nT s � �� n where \delta is the impulse function and T_s is the sampling period. The running index n is used with \delta to define the impulse train of the comb function. We now have: � � � � � � � g s x g x s x

Sampling (III) � � � � � � � G f G f * S f Taking the Fourier transform of g s (x) yields s � � � � � � � � � � � G f * f f nf � � s s � � � �� n � � � � � � f G f nf s s � �� n Where f s is the sampling frequency and * denotes convolution. We make use of the following properties of the Fourier Transform 1. Multiplication in the spatial domain corresponds to convolution in the frequency domain. 2. The Fourier transform on an impulse train is itself an impulse train. 3. The spectrum of a signal sampled with frequency f s (T s =1/f s ) yields the original spectrum replicated in the frequency domain with period f s |G s (f)| -f max f max -f s f s

Sampling (IV) The above result reveals that the sampling operation has left the original input spectrum intact, merely replicating it periodically in the frequency domain with a spacing of f_s. This allows us to rewrite G_s(f) as a sum of two terms, the low frequency (baseband) and high frequency components. The baseband specturm is exactly G(f), and the high frequency components, G_high(f), consist of the remaining replicated versions of G(f). G_s(f) = G(f) + G_high(f) Exact signal reconstruction from sampled data requires us to discard the replicated spectra G_high(f), leaving only G(f), the spectrum of the signal we seek to recover.

Reconstruction Conditions The only provision for exact reconstruction is that G(f) be undistorted due to overlap with G_high(f). Two conditions must hold for this to be true: 1. The signal must be bandlimited. This avoids spectra with infinite extent that are impossible to replicate without overlap. (In most imaging systems, the psf ensures this condition is satisfied in practice). 2. The sampling frequency f_s must be greater than twice the maximum frequency f_max, present in the signal. This minimum sampling frequency, known as the Nyquist rate, is the minimum distance between spectra copies, each with bandwidth f_max. f_s>f_nyquist=2*f_max

The Ideal Low-Pass Filter Given that it is theoretically possible to perform reconstruction, how may it be done? The act of reconstruction requires us to completely suppress G_high(f). This is done by multiplying G_s(g) with H(f) given as: � � 1 f f � � � max H f � � 0 f f � max |G s (f)| H(f) -f max f max -f s f s

Sinc Function In the spatial domain, the ideal low-pass filter is derived by computing the inverse Fourier transform of H(f). This yields the � � � sin x sinc function defined as: � � � sinc x � x Since multiplication in the frequency domain is identical to convolution in the spatial domain, then we can convolve the sample signal g_s(x) with sinc(x) to obtain the continuous g(x). � � � � � � � � � � � � � � � � � � g x sinc x * g x � sinc g x d s s � The problem is that sinc(x) requires infinite support which is impossible in practice.

Ringing effect When using a truncated sinc function for reconstruction we get ringing. These artifacts, known as the Gibbs phenomenon, are the overshoots and undershoots caused by reconstructing a signal with truncated frequency terms. Approximations to the ideal low pass filter balance between tampering frequencies below f_max and not suppressing fully frequencies above it.

Nonideal Reconstruction Since H(f), the ideal filter cannot be computed in practice, we resort to nonideal filters. � This bring up the problem of how to assess the quality of a filter |G(f)| H r (f) -f max f max -f s f s H_r(f) attenuate some frequencies that should remain intact and pass and pass some of the high frquency that should be suppressed. This leads to the problem of aliasing: aliasing: signals “traveling in disguise” as other frequencies high frequencies disguised as low frequencies

Aliasing aliasing : signals “traveling in disguise” as other frequencies It can be solved in one of two ways: 1. Increasing sampling rate (to reach Nyquist rate) 2. Applying low pass filter (to bandlimit the signal)

Aliasing in video Slide by Steve Seitz