Online Classification of Photo-Realistic Computer Graphics & Photographs – Lessons Learned Tian-Tsong Ng, Shih-Fu Chang Digital Video and Multimedia (DVMM) Lab Columbia University, New York, USA Mao-Pei Tsui Department of Mathematics University of Toledo, Ohio, USA
Image/Video Forensics Seeing is Believing? London Bombing 05’ Tsunami 05’ Bangkok Coup 06’ Hall of Fame of Image Forgery Internet Scientific Journal News: LA Times Images from www.camerairaq.com/faked_photos/ and www.worth1000.com
Related Problem: Image/Video Source Identification � Are multiple videos of the same event captured by the same source? � Are the visual imageries from real-world events or synthesized by advanced graphics tools? Graphics Or Photo? From same camera? Alias 3D design Two video shots from a CNN new topic (Alias fake_or_foto site)
Columbia TrustFoto System (www.ee.columbia.edu/trustfoto) Forensics Criminal Insurance Surveillance Intelligence Financial Journalism Investigation Investigation Processing video Services Industry With Expert Intervention Camera camera I D Images Natural Image Sensor Tampering Statistics Signature Artifact Modeling Extraction Detection Suspicious Regions Signal Processing Computer Graphics Approach Computer Graphics Inconsistent Approach Shadows Tampered Images With Photoshop Report Smoothing 3D Geometry Inverse 3D Scene Splicing Reconstruction Rendering Consistency Sharpening Checking
Principles: Joint Physics and Statistics 1. Use physics features derived from real- world scenes Image/video 3. Find manipulation Generation artifacts to detect splicing Process: 2. Recover camera features from images Joint Physics-based & Statistical Method:
Typical Camera Response Function (CRF)? Scene radiance R = f (r) Image Image Intensity Irradiance Typical CRF R(x; y) r(x; y) f (¢ ) Camera sensor
Estimate Camera Response Function (CRF) from a Single Image [Ng & Chang CVPR 07] � From locally planar regions, derivative ratios contains unique camera (CRF) information Locally R(x,y) CRF r(x,y) planar R y y r x x Find the best curve to fit the measured invariant features
(Hsu & Chang ’06) Camera Signature (CRF) consistency � CRF checking in real broadcast videos Different Sources: CNN vs. ABC Different CRFs 85.9% accuracy over Columbia dataset Consistent CRFs in all color channels confirms same source Images from TRECVID data set
Double quantization artifacts from splicing He, Lin, et al ECCV 06 Compression Re- Cut & paste /quantization Quantization Unique DCT coeff. patterns after double quantization Detecting spliced regions by detecting the unique artifacts
Example results From He, Lin, et al ECCV 06
A Physics-based Approach to Classify Photo vs. CG [Ng, Chang, Tsui ’05] � Analyze the physical differences between Photo and CG, in terms of the image generative process. � Propose a geometry-based image description framework Image Surface Geometry
Image Generative Process � Photographic Images (3) Non-linear camera response function - Not an arbitrary transform. (1) Complex surface model Light source - Subsurface scattering of human skin. - Color dependency. (2) Complex object geometry - Human skin texture follows biological system. - Building surface formed by natural erosion.
Image Generative Process � Computer Graphics Differences between Photo and CG (3) Non-standard Post-processing - Subject to the artist’s taste. Light source - Different from camera transform. (1) Simplified surface model - Assume color independence. Post-processing (2) Polygonal object geometry - Reduced mesh resolution for computational efficiency. - Without care, it introduces unnatural structures in rendered images.
Feature Correspondences Surface Model Difference Object Model Difference Acquisition Difference Differential Surface Second Fundamental Image Form Geometry Laplacian Gradient Fractal Distribution of the Local Geometry Fractal Dimension Local Distribution of the Local patches Patch Statistics
Typical Camera Response Function (CRF)? Scene radiance R = f (r) Image Image Intensity Irradiance Typical CRF R(x; y) r(x; y) f (¢ ) Camera sensor
Differential Geometry I Image Gradient Non-linear camera transform has effects on image Gradient ! � Low Irradiance High Irradiance Camera Model Chain Rule dr r dr dx r image irradiance dx dr dx df R=f(r) dr Camera Transfer df Slope of the Function curve dr Compress Expand dR R dR df dr = dx image Intensity dx dr dx
The Visual Effect of CRF Transform
Differential Geometry II Second Fundamental Form � Polygonal Model leads to unsmooth structures � At the junctures, the polygon is always sharper than the smooth curve. A smooth curve is approximated by a polygon Unusually sharp transition
Differential Geometry II Second Fundamental Form Locally, any surface can be written as a graph of a differentiable � function over the tangent plane. The local graph can be approximated by a quadratic function. � The Hessian of the quadratic function is the second fundamental form. � The Hessian can be characterized by 2 eigenvalues � Large eigenvalues implies sharp structures � 3D plot of elliptic Quadratic function. Cross-section of the quadratic function at z=1. (2,1) (3,1) eigenvalues (1,1)
Differential Geometry III Surface Laplacian � Rendering of CG often assumes color independence in the object surface model (generally, not true for real- world object): � We capture the difference in the RGB correlation for Photo and CG using the surface Laplacian. (R,G,B) � Laplacian operator ( Δ g ) ( Δ g I) = ( Δ g I R , Δ g I G , Δ g I B ) on a graph surface � A vector pointing to the direction which decreases the surface area. � For a submanifold in the y 5D space, it measures the correlation between R, G 5D Euclidean x and B. Space
Differential Geometry III Surface Laplacian Misalignment with 45 deg line 45 deg line 20% of CG has this misalignment, compared to only 5% of Photo.
Local Patch Statistics [Lee et al. 2003] 3x3 local patch forms a 2D sub-manifold in the � normalized 8D Euclidean space. [Rosales et al. 2003] Use local patches to characterize image styles � (e.g., Van Gogh Style). Patch dictionary from a Van Gogh Image. translation Input Photo Van Gogh style Image Photo and CG are just images of different styles! �
Fractal Geometry Surface property of the real-world objects may be modeled by � the fractal geometry. Fractal dimension measures the factor of self-similarity across � scales Fractional Brownian Motion model for images: � t: scale Fractal Dim. = (road) (tree) 0.5 - slope The faster the 2 nd - order differential decreases, the higher fractal dim is.
Recap: Physics-based feature pool (f) (p) (g) (s) (b) compute local features Compute features of point image over each distributions pixel or local (e.g. rotational moments) patch
Effectiveness of the features � Gaussian plots in 2D projection space � Confirms discriminativeness of the proposed features Red = Photo Blue = CG Surface Laplacian Second Fundamental Gradient (Beltrami) Form
Dataset Columbia Open Dataset First publicly available Photo/CG dataset, downloaded 20+ groups � Consists of 4 subsets, 800 images for each subset. � From a few Recaptured Personal Google Internet personal CG Photo Photo CG collections of photo Recaptured from Downloaded from Downloaded from the a LCD screen by Google Image Search 3D artist websites a Canon G3 camera Available at http://www.ee.columbia.edu/trustfoto
Test Set Covers Diverse Conditions
Comparison with Other Work in Photo vs. Photorealistic CG Classification [Ianeva et al. 03] Classifying photo and general CG (including � drawing and cartoon). Use simple color distributions, intensity, edge features. � [Lyu & Farid 05] Classifying photo and photorealistic CG. � Use image statistics from wavelet coefficients. � 67% detection rate (1% false alarm). � Lack strong insight into the physical differences between photo and CG. � [Wang & Moulin 06] Classifying photo and photorealistic CG. � Based on the marginal distributions of the wavelet coefficients. � Capture the difference using characteristic functions of distributions. � On a different dataset: 100% detection rate (1% false alarm). �
Wavelet Higher-order Statistics Features [Lyu & Farid ’05] Compute the mean, variance, skewness and kurtosis of the 72 dims coefficients for each subband Predict the Green coefficient from Red coefficients, and compute the prediction error. Compute the mean, variance, skewness and kurtosis of the 72 dims prediction errors.
Experimental Results I Support Vector Machine Classification SVM classification with radial basis function (RBF) kernel. � Features Geometry Wavelets Cartoon Accuracy 83.5% 80.3% 71.0% Receiver Photo (Detection rate) operating Vs characteristic Internet CG (ROC) curve (false alarm)
Recommend
More recommend
