Smartphone Imaging Trends Brian Klug Sr. Smartphone Editor, - PowerPoint PPT Presentation

Smartphone Imaging Trends Brian Klug Sr. Smartphone Editor, AnandTech.com Thursday, February 7, 13

Background B.S. - Optical Sciences & Engineering, University of Arizona Imaging Technology Lab with Steward Observatory Worlds first curved front-illuminated CCD ITL does BSI processing and characterization for CCDs used in astronomy, other photometric fields Thesis/Capstone: Terahertz GRIN Rapid prototyping First THz GRIN objective cheaply fabricated on 3D printer Thursday, February 7, 13

Background AnandTech.com , Founded in 1997 by Anand Lal Shimpi Smartphones to Servers and Everything in Between Everything is a Computer Strong Background and Emphasis on Components Understand the Pieces to Understand the Pie Thursday, February 7, 13

Optics 101 Index of refraction - n (unitless) Material property - ratio of how much the speed of light is slowed in a medium Wavelength - λ (m), Frequency - ν (Hz) Essentially “Color”, Human Response 400 - 700 nm Focal Length - f (m), Power - ϕ (diopters) Convergence or divergence of light from a system Longer focal length - more magnification, lower focal length, less magnification Thursday, February 7, 13

Optics 101 F-number - F/# (unitless), F/#=f/d Describes the size of the cone of light accepted by the system / light collection ability. Lower F/# - more light, equal to the ratio of the focal length to diameter of the entrance pupil F-Stops - typically go in √ 2 steps (1.4, 2, 2.8, 4, 5.6) which changes light collection by factor of 2 Optical / Image Sensor Format - eg. 1/3.2” (inches) Sensor size, but nothing to do with sensor size. Originally vidicon glass tube diameter required for some other active imaging area size. Use table. Thursday, February 7, 13

Optical Systems Many different optical systems Rifle Scopes Telescopes Microscopes Viewfinders Illumination / Projection Industrial / Science Internet (Fiber/EDFA) Thursday, February 7, 13

Camera Systems Approximation of the human eye Human eye images field onto retina using crystalline lens which changes index as stretched (focus) Objective system Form an image of a scene onto some plane, image a distant object, hence objective Simple Objective Telephoto Objective Thursday, February 7, 13

Imaging Block Diagram UI/OS/ Image CMOS Scene Optics Saved Signal Sensor Processor JPEG Thursday, February 7, 13

Video Block Diagram Same fundamental architecture, but with either a crop of the sensor or decimated version of output, then through an encoder for H.264/MPEG4. Encoder usually on SoC. CMOS Video Video Scene Optics ISP Encoder Sensor .mp4 Thursday, February 7, 13

Smartphone Context Smartphone camera systems have unique constraints Very small throw (z-stack, module often thickest part) Cost ($5-15 for module) Limited materials (Almost always plastic) Unique manufacturing (Aspheres - injection molding) Horrible operating conditions (Every type of scene) Small aperture (Battling ID of phone) All while imaging onto tiny pixels (Impossible problem) Thursday, February 7, 13

Smartphone Optical System Optics usually made of plastic, injection molded into aspheres (complex, nonspherical shapes). Limited to 2-5 elements (2P-5P). Glass uncommon. Optical plastics quite limited: Styrene, Polystyrene, ZEONEX, PMMA(Acrylic) Doublet: PMMA as Crown Polystyrene as Flint Fixed focal length, fixed aperture (no iris), sometimes an ND filter, no shutter, usually not very fast (f/2.8, 2.4) Short focal length (wide), tiny image circle formed Thursday, February 7, 13

Example Lens List Thursday, February 7, 13

Example System “Lens system having wide-angle, high resolution, and large aperture” US 8320061 B2, Chun-Cheng Ko, Hon Hai Precision Industry Co., Ltd. (aka Foxconn) an aperture stop; a first lens of positive refractive power having a subject-side surface and an image-side surface; a second lens of negative refractive power having a subject-side surface and an image-side surface; a third lens of positive refractive power having a subject-side surface and an image-side surface; and a fourth lens of negative refractive power having a subject-side surface and an image-side surface; Thursday, February 7, 13

Example Prescription Conic Radius of Curvature Index at d Constant (k) Separation Abbe # at d Polystyrene http://refractiveindex.info/? group=PLASTICS&material=PS Zeronex http://refractiveindex.info/? group=PLASTICS&material=ZeonexE48R Thursday, February 7, 13

Example 5P System (LG) Final Assembly First Element Last Element Thursday, February 7, 13

Lens then goes into a module Thursday, February 7, 13

Camera Module Lens assembly VCM (Voice Coil Motor) - electromagnet / speaker which moves the lens to focus IR Filter / AA filter CMOS sensor Packaging and ribbon flex cable Drop that module into a phone Thursday, February 7, 13

Imaging Block Diagram UI/OS/ Image CMOS Scene Optics Saved Signal Sensor Processing JPEG Thursday, February 7, 13

CMOS Sensor Trends Type Diagonal (mm) Width (mm) Height (mm) Area (mm 2 ) Crop factor 1/4" 4 3.2 2.4 7.68 10.81 1/3.6" 5 4 3 12 8.65 1/3.2" 5.68 4.54 3.42 15.5 7.61 1/3" 6 4.8 3.6 17.3 7.21 Rear CMOS size commonly around 1/3.2” or 1/4” Front CMOS smaller, but lower resolution 1/6”, 1/7” Size of CMOS sensors are relatively fixed, trend is more of smaller pixels Thursday, February 7, 13

CMOS Sensor Trends Generation n-3 n-2 n-1 n Future (?) Pixel Size ( µ m) 2.2 1.75 1.4 1.1 0.7 Area ( µ m^2) 4.84 3.06 1.96 1.21 0.49 Area Ratio 1.00 0.63 0.40 0.25 0.10 Waves, λ (@ 700 nm) ~3 ~2.5 ~2 ~1.5 ~1 Pixels : CMOS :: MHz : CPU - (MHz race, pixels) Pixels can’t get much smaller, or they’ll be sub one wave in size (weird quantum effects begin) >=5 MP (1.4µm), BSI is a necessity not just for sensitivity Thursday, February 7, 13

BSI / FSI FSI - Traditional way of imaging onto a CCD/ CMOS, through metal gating, incurring reflections BSI - Requires removing material using wafer scale chemical or abrasive lapping, image directly into active region of sensor. Significantly higher QE. Thursday, February 7, 13

Dirty Secret - IT’S A LIE We can’t resolve pixels that small to begin with... Oops.... from numpy import * from scipy import * from pylab import * f = 3.63E-3 ## 3.63 mm focal length HTC One system b = 1.4E-6 ## 1.4 micron standard modern pixel size, 1.1 newer, 1.65 micron older fno = 2.0 ## HTC One S F/2.0 fnos = linspace(1.2,2.8,100) ## HTC One - F/2.0, iPhone 4S - F/2.4, SGS3 - F/2.6 bscale = linspace(0.7E-6,1.65E-6,100) ## Range of pixel sizs zh = - f**2 / (b * fnos); ## hyperfocal distance zn = zh / fno ## nearest point in focus (hyperfocal/2) diff = (2.44 * 700E-9 * fnos)/1.0E-6 ## airy disk first zero diameter 84% energy here deltazprime = 4.88*700E-9 * fno ** 2 deltaz = (4 * deltazprime * f ** 2) / ((deltazprime)**2 - 4*(f - 3.7E-3)**2) cla() clf() plot(fnos,diff) title('Airy disk diameter as a function of F-Number') xlabel('F/#') ylabel('Spot size in microns') savefig('fnosspotsize.png') cla() clf() plot(fnos,zn) title('Hyperfocal distance as F/#') xlabel('F/#') ylabel('Hyperfocal dist') savefig('hyperfocalfixedpixel.png') cla() clf() zh2 = - f**2 / (bscale * fno); ## hyperfocal distance fixed f=2.0 plot(bscale/1.0E-6,zh2/2) title('Hyperfocal distance as pixel size') xlabel('Pixel size (microns)') ylabel('Hyperfocal dist') savefig('hyperfocalfixedfnos.png') Thursday, February 7, 13

Dirty Secret - IT’S A LIE Airy Disk - Assumes perfect optics, limited only by diffraction (ideal system) Rayleigh Criterion - camera example, before two points blur together Can’t resolve that pixel size! Oops. Thursday, February 7, 13

Hyperfocal distance Distance after which everything is in focus Thursday, February 7, 13

Imaging Block Diagram UI/OS/ Image CMOS Scene Optics Saved Signal Sensor Processing JPEG Thursday, February 7, 13

ISP - Image Signal Processor ISP usually onboard SoC, sometimes discrete ISP Roles Demosaicing - Sensor just senses photons, need Bayer color filter atop sensor to determine color. RGBG / GRGB, interpolate to RGB for each pixel. 3A - Autofocus, Autoexposure, Autowhitebalance Correction for lens imperfections - Lens shading, geometry/distortion, vignetting, try to fix image Noise reduction, filtering, HDR, cleaning up, JPEG This is the controller for CMOS / Focus assembly Thursday, February 7, 13

Imaging Block Diagram UI/OS/ Image CMOS Scene Optics Saved Signal Sensor Processing JPEG Thursday, February 7, 13

Camera UI/UX Minimalist to highly customizable Still evolving, Many still making horrible mistakes Low res/fps preview, wrong preview, broken UI, not enough controls, laggy Smartphone platform again unique - needs balance of speed and simplicity to be successful Thursday, February 7, 13

Video Block Diagram Same fundamental architecture, but with either a crop of the sensor or decimated version of output, then through an encoder for H.264/MPEG4. Encoder usually on SoC. CMOS Video Video Scene Optics ISP Encoder Sensor .mp4 Thursday, February 7, 13