Multimedia & Mathematics July 23-28, 2005 Banff, Alberta A Unified Framework for the A Unified Framework for the Consumer-Grade Image Pipeline Consumer-Grade Image Pipeline Konstantinos N. Plataniotis University of Toronto kostas@dsp.utoronto.ca www.dsp.utoronto.ca Common work with Rastislav Lukac Outline Outline • The problem • Background • Single Sensor Imaging: Challenges & Opportunities • Performance issues • Conclusions
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Digital color imaging Digital color imaging spatial position i = ( k − 1) K + k image column 1 1 2 i = image sample x ( 1 86 ,4 , 8 42 ) k x = 2 186 color image Parrots i 1 = x 48 i 2 R channel G channel B channel x = 42 i 3 image row k 1 K 1 (number of image rows) l = 2 K (image dimension) 2 RGB image R G B RG image RB image GB image (number of image columns) Color image acquisition: m = 3 (number of color channels) • digital cameras - most popular and widely used • scanners • synthetic (e.g. gray-scale image coloration) Focusing Focusing on the color pixel level on the color pixel level RGB (sRGB) color space: Red • commonly used for acquisition, storage, and displaying purposes Magenta Yellow • additive concept of color composition White Red Magenta 1 Blue Green x i x j Cyan Yellow White • RGB color pixel is the vector in a three- Black Blue 0 dimensional (RGB) color space 1 • vector components are the intensities Maxwell measured in RGB color channels triangle 1 Green Cyan
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Color imaging basics Color imaging basics Color vector: uniquely characterized by its = = + + M x ( x ) 2 ( x ) 2 ( x ) 2 • magnitude (length) x i i 1 i 2 i 3 i x x x • direction (orientation) = = D i 1 , i 2 , i 3 ; D 1 x x M M M i i x x x i i i R R x i 1 1 M x unit sphere i 2 ( x ) i 2 x i 1 2 ( x ) x i 1 x i i 3 x 0 0 M x x i 2 D x x i M x i i i 3 i 2 ( x ) i 3 1 1 G x B G B i 2 Camera: End-user’s point of view Camera: End-user’s point of view Focus on effectiveness: functionality vs. cost • optics (optical zoom), digital zoom, memory, battery, etc. • multimedia acquisition, processing & transmission (image, audio and video)
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Three-sensor imaging Three-sensor imaging • “Sensor” : a monochromatic device; most expensive component of the digital camera (10% to 25% of the total cost) • charge-coupled device (CCD) • complementary metal oxide semiconductor (CMOS) sensor R filter + sensor optical G filter + system + sensor image camera B filter scene output + sensor color filters + sensor sensor data image sensors data arranged as • professional designs (CCD/CMOS) RGB color data • each sensor corresponds to a particular color channel • spectrally selective filters • expensive solution X3 technology-based single-sensor X3 technology-based single-sensor imaging imaging Layered (three-layer) silicon sensor • new technology - expensive solution for professional devices (medical & science applications) • directly captures RGB light at each spatial location in an image during a single exposure • takes advantage of the natural light absorbing characteristics of silicon • color filters are stacked vertically and ordered according to the energy of the photons absorbed by silicon
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Single-sensor imaging Single-sensor imaging sensor data optical CFA + demosaicking system sensor image color filter array sensor data camera + image sensor arranged as scene output (CCD/CMOS) RGB color data Color filter array (CFA) • generates a 2-D array or mosaic of color components • produced CFA (sensor) image is a gray-scale image • full-color image is obtained through digital processing Color Color filter array (CFA) design ilter array (CFA) design Key factors in CFA design • immunity to color artifacts and color moiré • cost-effective image reconstruction • reaction of the pattern to image sensor imperfections • immunity to optical/electrical cross talk between neighboring pixels Color systems used in CFA design i) tri-stimulus (RGB, YMC) systems - RGB is most widely used ii) mixed primary/complementary colors (e.g. MGCY pattern) iii) four and more color systems (white and/or colors with shifted spectral sensitivity) - • CFAs in ii) and iii) may produce more accurate hue gamut, but they limit the useful range of the darker colors
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Common RGB-based CFAs Common RGB-based CFAs Bayer CFA Diagonal stripe CFA Vertical stripe CFA Yamanaka CFA Pseudo-random CFA Pseudo-random CFA HVS based design Diagonal Bayer CFA • Bayer CFA is widely used (good performance, cost-effective color reconstruction) Single-sensor camera architecture Single-sensor camera architecture lens, zoom, focus infrared blocking, aperture and shutter anti-aliasing optical filter viewfinder optical blocking (Bayer) image sensor image system system CFA (CCD,CMOS) scene A/D stick memory converter flash media (card) bus user micro- camera color processor ASIC interface LCD display power supply PC / TV interface DRAM firmware (battery, AC) (USB, A/V) buffer memory • DRAM buffer temporally stores the digital data from the A/D converter • DRAM then passes data to the application-specific integrated circuit (ASIC) • digital data processing, such as demosaicking and image resizing, is realized in both ASIC and microprocessor
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Camera image processing Camera image processing � Processing • demosaicking (spectral interpolation) • demosaicked image postprocessing (color image enhancement) • camera image zooming (spatial interpolation in CFA or full-color domain) Compression • lossy (or near lossless) vs. lossless compression • CFA image compression vs. demosaicked image compression Data management • camera (CFA) image indexing → connection to image retrieval Implementation Implementation Conventional digital camera • real-time constraints (computational simplicity requirements) camera image CFA data storage processing digital camera Using a companion personal computer (PC) • PC interfaces with the digital camera which stores the images in the raw CFA format • allows for the utilization of sophisticated solutions camera image storage CFA data storage processing digital camera personal computer (PC)
Multimedia & Mathematics July 23-28, 2005 Banff, Alberta Camera processing operations Camera processing operations Considering the spectral image characteristics • component-wise (marginal) processing (component → component) • spectral model-based processing (vector → component) • vector processing (vector → vector) Considering the image content (structure) • non-adaptive processing • (data) adaptive processing Edge-sensing Spectral Practical solutions mechanism model • spectral model used to eliminate color shifts and artifacts input camera estimation outputted • edge-sensing mechanism image operations camera image used to eliminate edge-blurring and generalized camera solutions to produce sharply-looking fine details Considering the spectral Considering the spectral characteristics characteristics Component-wise processing camera image processing • each color plane processed separately camera image processing • omission of the spectral information camera image results in color shifts and artifacts input color output color processing image image Spectral model based processing camera image processing • essential spectral information utilized camera image during processing processing • computationally very efficient - most camera image input color output color processing widely used in camera image processing image image Vector processing camera image • image pixels are processed as vectors processing • computationally expensive input color output color image image
Recommend
More recommend
