Mobile Image Retargeting Daniel Graf Bachelor Thesis February 28, - PowerPoint PPT Presentation

Mobile Image Retargeting Daniel Graf · Bachelor Thesis February 28, 2013 1

Image Retargeting February 28, 2013 Daniel Graf # 2

Image Retargeting crop center February 28, 2013 Daniel Graf # 2

Image Retargeting crop center scaling February 28, 2013 Daniel Graf # 2

Image Retargeting crop center scaling letterbox February 28, 2013 Daniel Graf # 2

Image Retargeting retargeting February 28, 2013 Daniel Graf # 3

Image Retargeting • content-aware • objectives ‣ preserve important structure and content ‣ no artifacts retargeting February 28, 2013 Daniel Graf # 3

Image Retargeting • content-aware saliency map • objectives ‣ preserve important structure and content ‣ no artifacts February 28, 2013 Daniel Graf # 4

Image Retargeting • content-aware artifacts • objectives ‣ preserve important structure and content ‣ no artifacts February 28, 2013 Daniel Graf # 5

Goals • bring image retargeting to a mobile device ‣ limited CPU, GPU and screen size • intuitive user interface for small displays • combine cropping and warping techniques February 28, 2013 Daniel Graf # 6

Related Work

Seam Carving • Seam Carving repeatedly remove the least salient path „Seam carving for content-aware image resizing.“ Avidan and Shamir, SIGGRAPH 2007 February 28, 2013 Daniel Graf # 8

Warping Techniques • look for a continuous mapping F : I → I 0 I 0 p 0 I p H 0 H W W 0 February 28, 2013 Daniel Graf # 9

Warping Techniques • look for a continuous mapping F : I → I 0 I 0 p 0 I p H 0 H W W 0 February 28, 2013 Daniel Graf # 9

Warping Techniques • look for a continuous mapping F : I → I 0 • modelled as energy minimization ✓ 1 ◆ 0 || 2 S ( x, y ) || J F ( x, y ) − 0 1 Jacobian of saliency map the mapping February 28, 2013 Daniel Graf # 10

Warping Techniques • look for a continuous mapping F : I → I 0 • modelled as energy minimization ✓ 1 Z H Z W ◆ 0 || 2 dxdy E ( F ) = S ( x, y ) || J F ( x, y ) − 0 1 x =0 y =0 Jacobian of saliency map the mapping February 28, 2013 Daniel Graf # 10

Warping Techniques • discretization on regular grid original grid deformed grid N columns G 0 G F M rows February 28, 2013 Daniel Graf # 11

Possible Discontinuities • impose hard constraints • leads to a convex optimization problem • optimization is expensive and only feasible on coarse grids February 28, 2013 Daniel Graf # 12

Possible Discontinuities • impose hard constraints • leads to a convex optimization problem • optimization is expensive and only feasible on coarse grids February 28, 2013 Daniel Graf # 12

Our Retargeting Operator

Axis-Aligned Image Deformations • linear number of variables s = ( s rows , s cols ) T • fast optimization s cols j +1 G 0 G F s rows i +1 „Robust image retargeting via axis-aligned deformation.“ Panozzo et al., EG 2012 February 28, 2013 Daniel Graf # 14

ASAP Energy • locally correspond to a similarity transform important cells should not change their aspect ratio February 28, 2013 Daniel Graf # 15

ASAP Energy • locally correspond to a similarity transform ◆◆ 2 ✓ ✓ M − N H s rows W s cols Ω i,j i j change in average saliency aspect ratio per grid cell • we use a grid of 25 x 25 cells February 28, 2013 Daniel Graf # 16

ASAP Energy • locally correspond to a similarity transform M N ◆◆ 2 ✓ ✓ M − N X X H s rows W s cols Ω i,j E ASAP = i j i =1 j =1 change in average saliency aspect ratio per grid cell • we use a grid of 25 x 25 cells February 28, 2013 Daniel Graf # 16

Constraints • row and column sums are target image size s row + s row + · · · + s row = H 0 M 1 2 s cols + s cols + · · · + s cols = W 0 1 2 N I 0 I H 0 H W W 0 February 28, 2013 Daniel Graf # 17

Constraints • no grid cell disappears: minimum cell size ≥ ¯ s rows H min , i = 1 , . . . , M i i ≥ ¯ s cols W min , j = 1 , . . . , N j j February 28, 2013 Daniel Graf # 18

Constraints • no grid cell disappears: minimum cell size ≥ ¯ s rows H min , i = 1 , . . . , M i i ≥ ¯ s cols W min , j = 1 , . . . , N j j February 28, 2013 Daniel Graf # 18

DEMO February 28, 2013 Daniel Graf #19

Automatic Saliency Detection

Automatic Saliency Detection • What is important? • Low-level stimuli driven approaches ‣ regions of high contrast ‣ we use gradient magnitude with dilation February 28, 2013 Daniel Graf # 21

Automatic Saliency Detection • What is important? • Low-level stimuli driven approaches ‣ regions of high contrast ‣ we use gradient magnitude with dilation • High-level object based saliency ‣ we use the iOS face detector February 28, 2013 Daniel Graf # 22

User Interface

Problem February 28, 2013 Daniel Graf # 24

Problem • small screen does not allow split view February 28, 2013 Daniel Graf # 24

Problem • small screen does not allow split view • combine saliency map and retargeted image February 28, 2013 Daniel Graf # 24

Problem • small screen does not allow split view • combine saliency map and retargeted image • updated warp moves image under the finger I 0 I 0 1 2 p 1 p 1 p 2 February 28, 2013 Daniel Graf # 24

Fixed-point stabilization • move the image to fix the point at the finger I 0 I 0 2 1 p 2 p 1 February 28, 2013 Daniel Graf # 25

Fixed-point stabilization February 28, 2013 Daniel Graf # 26

Cropping

Threshold-based Cropping 1. optimize February 28, 2013 Daniel Graf # 28

Threshold-based Cropping 1. optimize 2. crop ✁ ✁ crop threshold February 28, 2013 Daniel Graf # 28

Threshold-based Cropping 1. optimize 2. crop ✁ ✁ crop threshold February 28, 2013 Daniel Graf # 28

Threshold-based Cropping 1. optimize 2. crop 3.optimize ✁ ✁ crop threshold February 28, 2013 Daniel Graf # 28

Threshold Range ✁ ✁ ✁ ✁ ✁ February 28, 2013 Daniel Graf # 29

Results

RetargetMe Benchmark original our operator CR SV MULTIOP AAIR February 28, 2013 Daniel Graf # 31

Results • interactive performance of 60 fps • cropping operator ‣ greediness of threshold approach • application ‣ picture gallery February 28, 2013 Daniel Graf # 32

Results • interactive performance of 60 fps • cropping operator ‣ greediness of threshold approach • application ‣ picture gallery February 28, 2013 Daniel Graf # 32

Future Work • more sophisticated cropping approaches • application in a (mobile) web browser • video retargeting ETH Zurich History E T H w a s founded in 1854 ETH Zurich by the Swiss Confederation History and opened its doors in 1855 as a polytechnic i n s t i t u t e ( E i d g e n ö s s i s c h e ETH was founded in 1854 by the Polytechnische Schule ). It comprised Swiss Confederation and opened in the beginning six departments: its doors in 1855 as a polytechnic institute ( Eidgenössische Polytechnische architecture, civil engineering, Schule ). It comprised in the beginning six departments: architecture, m e c h a n i c a l e n g i n e e r i n g , civil engineering, mechanical engineering, chemistry, forestry, and a chemistry, forestry, and a catch-all catch-all department for mathematics, natural sciences, literature, and department for mathematics, social and political sciences. It is locally still known as Poly , derived natural sciences, literature, and from the original name Eidgenössisches Polytechnikum "Federal social and political sciences. Polytechnic Institute". February 28, 2013 Daniel Graf # 33

Conclusion • fast, fluid, fun to use application • cropping operator is simple yet effective • next steps ‣ make source code publicly available ‣ submit app to the iOS App Store February 28, 2013 Daniel Graf # 34

Questions

Thank you for your attention.