In the name of Allah In the name of Allah the compassionate, the merciful
Digital Video Processing S. Kasaei S. Kasaei Room: CE 307 Department of Computer Engineering Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu Web Page: http://sharif edu/ Web Page: http://sharif.edu/~skasaei skasaei http://ipl.ce.sharif.edu
Chapter 3 Chapter 3 Video Sampling Video Sampling
Vid Video Sampling Main Concerns S mplin M in C n rn 1. What are the necessary sampling at a e t e ecessa y sa p g frequencies in the spatial & temporal directions? 2. Given an overall sampling rate ( i.e., product of the horizontal, vertical, & temporal sampling rates), how do we sample in the 3-D space to obtain the best representation? best representation? 3. How can we avoid aliasing? Kasaei 5
Video Sampling (A Bri f Di (A Brief Discussion) i n) � Review of Nyquist sampling theorem in 1-D � Review of Nyquist sampling theorem in 1 D � Extension to multi-dimensions � Prefiltering in video cameras g � Interpolation filtering in video displays Kasaei 6
Nyquist Sampling Theorem in 1 D in 1-D � Given a band-limited signal with maximum g frequency f max , it can be sampled with a sampling rate f s >=2 f max . � The original continuous signal can be exactly � The original continuous signal can be exactly reconstructed (interpolated) from the samples, by using an ideal low pass filter with cut-off frequency at f s /2. at f s /2. � Practical interpolation filters: replication (sample-and-hold, 0 th order), linear h interpolation (1 st order), cubic-spline (2 nd order). ) Kasaei 7
Nyquist Sampling Theorem in 1 D in 1-D � Given the maximally feasible sampling rate f s , � Given the maximally feasible sampling rate f s , the original signal should be bandlimited to f s /2, to avoid aliasing. � The desired prefilter is an ideal low-pass filter with cut-off frequency at f s /2. � Prefilter design: Trade-off between aliasing & loss of high frequency content. g q y Kasaei 8
Extension to Multi-Dimensions E t n i n t M lti Dim n i n � If the sampling grid is aligned in each � If the sampling grid is aligned in each dimension (rectangular in 2-D), and one performs sampling in each dimension separately, the extension is straightforward: t l th t i i t i htf d � Requirement: f s,i <= f max,i /2. � Interpolation/prefilter: ideal low pass in each � Interpolation/prefilter: ideal low-pass in each dimension. Kasaei 9
B Basics of Lattice Theory i f L tti Th r Λ Λ k � A lattice, , in the real K -D space, ,is � A lattice, , in the real K D space, ,is R R the set of all possible vectors that can be represented as integer-weighted combinations of a set of K linearly independent basis vectors, that is: K ∑ Λ = ∈ = ∀ ∈ k x R | x n v , n Z k k k = k k 1 with generating matrix: [ ] [ [ ] [ ] ] = V V v v , , , v L 1 2 k Kasaei 10
Basis Vectors Rectangular g Hexagonal g Lattice Lattice Lattice Points Reciprocals Voronoi Cell Kasaei 11
B Basics of Lattice Theory i f L tti Th r � One can find more than one basis or � One can find more than one basis or generating matrix that can generate the same lattice. � Given a lattice, one can find a unit cell , such that its translations to all lattice points form a tiling of the entire space. Kasaei 12
L tti Lattice Unit Cells Unit C ll Unit Cells Kasaei 13 (in spatial domain)
L tti Lattice Unit Cells Unit C ll � Unit cells are of two types: fundamental � Unit cells are of two types: fundamental parallelepiped & Voronoi cell. � There are many fundamental parallelepipeds Th f d t l ll l i d associated with a lattice (because of the nonuniqueness of the generating matrix). � The volume of the unit cell is unique. � A hexagonal lattice is more efficient than a rectangular lattice (as it requires a lower sampling density to obtain an alias-free li d it t bt i li f sampling). Kasaei 14
V r n i C ll D t rmin ti n Voronoi Cell Determination Determination of Voronoi cell: � Draw a straight line between the origin & each one of the closest nonzero lattice points. l tti i t � Draw a perpendicular line that is the half way between the 2 points. � This line is the equidistance line q between the origin & this lattice point. Kasaei 15
R Reciprocal Lattice ipr l L tti Λ Λ * � Given a lattice, its reciprocal lattice, , is defined , p , , as a lattice that its basis vector is orthonormal to that of the lattice. [ ] [ ] = − or =I T U T 1 V U ( V ) � The denser the lattice the sparser its reciprocal � The denser the lattice, the sparser its reciprocal. � A generalized Nyquist sampling theory exists, g yq p g y , which governs the necessary density & structure of the sampling lattice for a given signal spectrum spectrum. Kasaei 16
S mplin Sampling over Lattices r L tti � Fourier transform of a sampled signal over ou e t a s o o a sa p ed s g a o e a lattice is called the sample-space Fourier transform (SSFT). � SSFT reduces to DTFT when the lattice is a hypercube hypercube. � That is when [ V ] is a K -D identity matrix. � SSFT is periodic with a periodicity matrix [ U ]. Kasaei 17
S mplin Sampling over Lattices r L tti � To avoid aliasing, the sampling lattice � To avoid aliasing, the sampling lattice must be designed so that the Voronoi cells of its reciprocal lattice completely cover the signal spectrum. � To minimize the sampling density it � To minimize the sampling density, it should cover the signal spectrum as tightly as possible. tightly as possible. Kasaei 18
S mplin Sampling over Lattices r L tti � Most real-world signals are symmetric in � Most real world signals are symmetric in frequency contents (spherical support). � Interlaced scan uses a non-rectangular lattice in the vertical-temporal plane. p p Kasaei 19
Input Sampled Signals Signals Kasaei 20
S mplin Effi i n Sampling Efficiency d Λ d Λ ( ) ( ) Sampling density: , # of lattice points p g y , p ρ Λ ( ) Sampling efficiency: Kasaei 21
S mplin Sampling of Video Signals f Vid Si n l � Most motion picture cameras sample a scene in p p the temporal direction. � Store a sequence of analog frames on a film. � Most TV cameras capture a video sequence by sampling it in temporal & vertical directions. sampling it in temporal & vertical directions. � 1-D raster scan. � To obtain a full digital video, one should: � Sample analog frames in 2-D. � Sample analog raster scan in 1-D. p g � Acquire discrete video frames directly using a digital camera (by sampling a scene in 3-D). Kasaei 22
Required Sampling Rates R q ir d S mplin R t � Sampling frequency (frame rate & line rate): � Sampling frequency (frame rate & line rate): � Frequency content of the underlying signal. � Visual thresholds in terms of the spatial & temporal cut-off frequencies. � Capture & display device characteristics. � Affordable processing, storage, & transmission Aff d bl i t & t i i costs. Kasaei 23
Interlaced Scan Progressive Sampling Sampling S Scan Lattice Sampling Lattice Nearest Aliasing Components Kasaei 24
S mplin Vid Sampling Video in 2-D in 2 D 1. The same 2-D sampling density. e sa e sa p g de s ty 2. The same 2-D nearest aliasing. 3. Different nearest aliasing along the temporal frequency axis. Less flickering for interlaced. � 4. Different mixed aliases. N earest off-axis alias component. � 5. For a signal with isotropic spectral For a signal with isotropic spectral 5 support, the interlaced scan is more Kasaei 25 efficient.
Generating Sampling Matrix for Lattice Sampling Lattice Lattice Reciprocal Lattice Kasaei 26
Filt rin Op r ti n Filtering Operations � How practical cameras & display devices o p act ca ca e as & d sp ay de ces accomplish the required prefiltering & reconstruction filters in a crude way. � How the HVS partially accomplishes the required interpolation task. � Camera apertures consists of: � Temporal aperture. � Temporal aperture. � Spatial aperture. � Combined aperture. Kasaei 27
C m r Ap rt r Camera Apertures � Temporal aperture: � Temporal aperture: � Intensity values read out at any frame instant are not the sensed values at that time. � Rather they are the average of the sensed signal over a certain time interval, known as exposure time. � Camera is applying a prefilter in the temporal domain, called temporal aperture function . � Modeled by a low pass filter: h h , t ( t ( ) ) p t Kasaei 28
C m r Ap rt r Camera Apertures � Spatial aperture: � Spatial aperture: � Intensity value read out at any pixel is not the optical signal at that point alone. � Rather it is a weighted integration of the signal in a small window surrounding it, called aperture. � Camera is applying a prefilter in the spatial domain, called spatial aperture function . � Modeled by a circularly symmetric Gaussian function: h ( x , y ) p , x , y Kasaei 29
Recommend
More recommend
