2/8/2008 What’s coming? • Content ‐ aware retargeting Image and Video Retargeting • Texture synthesis CS 395T: Visual Recognition and Search Harshdeep Singh 16 4 3 9 What can be done? Resize Letterbox Crop 1
2/8/2008 Content ‐ aware Retargeting What is “Significant”? Lose the “insignificant” while preserving the • High energy regions “significant”… – Gradient, edges, entropy, histogram of gradient direction etc • High motion regions • High motion regions … and not disfiguring the image/video d di fi i h i / id – Or high motion contrast regions • Faces – Or other known objects like cars • Text Saliency Map Retargeting Algorithms Degree of saliency for each position in the image Retargeting Images Videos Warp Warp Crop based Crop based based based 1 2 3 4 Automatic Thumbnail Cropping Automatic Thumbnail Cropping • Problem – Find a rectangle in the image that • Has a small size • Contains most of the salient parts • Solution (Greedy) – Initialize Rc as a small rectangle at the center – While cumulative saliency < threshold • R = small rectangle around the next most salient point • Rc = Rc U R Automatic Thumbnail Cropping and its Automatic Thumbnail Cropping and its Effectiveness, Suh et al, 2003 Effectiveness, Suh et al, 2003 2
2/8/2008 Automatic Thumbnail Cropping User Experiments • Threshold can be adaptively chosen at the point of diminishing returns. • Finding sum of pixels in a rectangular area is very fast (Integral image/summed area tables) Automatic Thumbnail Cropping and its Automatic Thumbnail Cropping and its Effectiveness, Suh et al, 2003 Effectiveness, Suh et al, 2003 Seam Carving Use of Dynamic Programming Vertical seam – an 8 ‐ connected path of pixels from top to • To find the optimal seam bottom, containing one pixel in each row • To find the optimal order of horizontal and vertical seams to Horizontal seam – left to right be removed to resize an n x m image to n’ x m’. Remove lowest energy seam iteratively Energy of a pixel Seam Carving for Content ‐ Aware Image Seam Carving for Content ‐ Aware Image Resizing, Avidan et al, SIGGRAPH 2007 Resizing, Avidan et al, SIGGRAPH 2007 Works? Image Enlarging Find k lowest energy seams. Insert a new seam for each of them by averaging with left and right neighbors. Original Inserting k lowest energy seams Using IntuImage ‐ http://www.intuimage.com/ Conventional resizing Repeatedly inserting the same seam Seam Carving for Content ‐ Aware Image Seam Carving for Content ‐ Aware Image Resizing, Avidan et al, SIGGRAPH 2007 Resizing, Avidan et al, SIGGRAPH 2007 3
2/8/2008 Other applications Cropping vs. Warping Content amplification Scale up the image using standard methods. Apply seam carving to bring back to original dimensions. Object removal User marks an object. Remove seams until all marked pixels have been eliminated. Insert new seams. Seam Carving for Content ‐ Aware Image Image: Non ‐ homogeneous Content ‐ driven Resizing, Avidan et al, SIGGRAPH 2007 Video ‐ retargeting, Wolf et al, ICCV 2007 Video Retargeting by Cropping Video Shot Detection • Salient region may change from one frame to another • Shot – An unbroken sequence of frames from one • May need to add camera motion to preserve it camera • Detecting shot boundaries • Detecting shot boundaries • The resulting video must be cinematically plausible . – Pixel differences (Avoid zooms, instant camera acceleration etc) – Histogram comparisons • Works on each shot separately – Edge differences – Motion vectors Video Retargeting: Automating Pan and Comparison of video shot boundary Scan, Liu et al ,ACM Multimedia, 2006 detection techniques, Boreczky 1996 Retargeting one shot Retargeting video shot • Virtual Pans • Crop – Salient region changes – Salient features stay within the same region throughout during the shot gradually the shot – Limited to a single – A single cropping window for the entire shot horizontal pan – No camera motion added – Easy in easy out Video Retargeting: Automating Pan and Video Retargeting: Automating Pan and Scan, Liu et al ,ACM Multimedia, 2006 Scan, Liu et al ,ACM Multimedia, 2006 4
2/8/2008 Retargeting video shot Video Retargeting by Warping • Warp – maps pixels in the original frame to the retargeted frame • Virtual Cuts – Salient region changes abruptly • An unimportant pixel should be mapped close to its neighbors – One shot into two – Gets blended with them – One subshot comes from the • An important pixel should be mapped far from its neighbors An important pixel should be mapped far from its neighbors left part, other from the right left part other from the right – Size of regions of important pixels remains the same Video Retargeting: Automating Pan and Non ‐ homogeneous Content ‐ driven Video ‐ Scan, Liu et al ,ACM Multimedia, 2006 retargeting, Wolf et al, ICCV 2007 Optimize under constraints Benefits over Seam ‐ Carving • Maintains temporal coherence in videos 1. Each pixel should be at a fixed distance from its left and right neighbors (depending on importance) • Causes less deformation under severe down ‐ sizing 2. Each pixel needs to be mapped to a location similar to one of its upper and lower neighbors 3. Mapping of a pixel at time t should be similar to its mapping at t+1 Original Wolf et al Seam ‐ Carving 4. Warped locations must fit to the dimensions of the target frame Non ‐ homogeneous Content ‐ driven Video ‐ Non ‐ homogeneous Content ‐ driven Video ‐ retargeting, Wolf et al, ICCV 2007 retargeting, Wolf et al, ICCV 2007 Texture Synthesis Roadmap • Goal – Create new samples of a given texture • A simple and intuitive algorithm But slow � • Many applications – virtual environments, hole filling, • • Efros and Leung, 1999 texturing surfaces • Acceleration strategies – Improving search time with a tree p g • Wei et al, 2000 – Synthesizing in bigger blocks, using spatial coherence • Efros and Freeman, 2001 • Video Textures • Schodl at al, 2000 • Using Graphcuts iteratively for image and video textures • Kwatra et al, 2003 Slide from Kristen, CS 378 Fall 07 5
2/8/2008 Efros & Leung ’99 Varying Window Size sampling p I Input image t i Synthesizing a pixel • Assuming Markov property, compute P(p|N(p)) – Building explicit probability tables infeasible – Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p – To synthesize p, just pick one match at random Increasing window size Slide from Efros SIGGRAPH 2001 Slide from Kristen, CS 378 Fall 07 Efros & Leung ’99 Multi ‐ resolution Pyramid • The algorithm Very simple High resolution Low resolution o Surprisingly good results o …but very slow o Bottlenecks • o Have to search entire input texture to synthesize each pixel o Bigger neighborhood => Slower search Fast Texture Synthesis using Tree ‐ structured Vector Quantization , Wei et al, Slide modified from Efros SIGGRAPH 2001 Slide from Wei, SIGGRAPH 2000 Tree ‐ structured Results Vector Quantization • Computation bottleneck: neighborhood search 1 level 1 level 3 levels 5 × 5 11 × 11 5 × 5 Fast Texture Synthesis using Tree ‐ structured Fast Texture Synthesis using Tree ‐ structured Vector Quantization , Wei et al, Vector Quantization , Wei et al, Slide modified from Wei, SIGGRAPH 2000 Slide from Wei, SIGGRAPH 2000 6
2/8/2008 Tree ‐ structured Comparison Vector Quantization Input Efros and Leung ‘99 Wei et al 1941 seconds 12 seconds Fast Texture Synthesis using Tree ‐ structured Fast Texture Synthesis using Tree ‐ structured Vector Quantization , Wei et al, Vector Quantization , Wei et al, Slide from Wei, SIGGRAPH 2000 Slide from Wei, SIGGRAPH 2000 block Efros & Leung ’99 extended Input texture B1 B2 B1 B2 B1 B2 p B Random placement p Neighboring blocks g g Minimal error I Input image t i of blocks constrained by overlap boundary cut Synthesizing a block • Observation: neighbor pixels are highly correlated Idea: unit of synthesis = block Idea: unit of synthesis = block • Exactly the same but now we want P(B|N(B)) • Much faster: synthesize all pixels in a block at once Slide from Efros SIGGRAPH 2001 Slide from Efros SIGGRAPH 2001 Minimal error boundary Synthesis Results overlapping blocks vertical boundary 2 _ = overlap error min. error boundary Slide from Kristen, CS 378 Fall 07 Slide from Efros SIGGRAPH 2001 7
2/8/2008 Texture Transfer Failures (Chernobyl • Take the texture from one object Harvest) and “paint” it onto another object – This requires separating texture and shape – That’s HARD, but we can cheat – Assume we can capture shape by Assume we can capture shape by boundary and rough shading Then, just add another constraint when sampling: similarity Then, just add another constraint when sampling: similarity to underlying image at that spot to underlying image at that spot Slide from Efros SIGGRAPH 2001 Slide from Efros SIGGRAPH 2001 Hole Filling parmesan + = = rice + = = Slide from Efros SIGGRAPH 2001 Slide from Kristen, CS 378 Fall 07 Video textures Finding good transitions Compute L 2 distance D i , j between all frames vs. frame i frame j Similar frames make good transitions Video Textures, Schodl et al, SIGGRAPH Video Textures, Schodl et al, 2000 Slide form Schodl, SIGGRAPH 2000 8
Recommend
More recommend
