Imaging Pipeline Instructors: Yasuhiro Mukaigawa, Takuya Funatomi, - PowerPoint PPT Presentation

Visual Media Processing II 2020 No.2 Imaging Pipeline Instructors: Yasuhiro Mukaigawa, Takuya Funatomi, Kenichiro Tanaka Today’s mini -report: http://bit.ly/vmp2-2020-2 Slide credit: Ioannis Gkioulekas, Tomokazu Sato

Today’s mini -report and Changes • Today’s mini -report http://bit.ly/vmp2-2020-2 • I’ve put the questions on the website. http://omilab.naist.jp/class/VMP2/2020/ Also, Google Forms contains full questions. • Strongly recommend to complete Mini-report in the class. Visual Media Processing II (4098), 2020 Fall 2

Photometric analysis pipeline Image processing / image analysis (#3 - #8) in-camera Optics Sensor processing (#1) (Today) Visual Media Processing II (4098), 2020 Fall 3

Digital Image Consists of Pixels Visual Media Processing II (4098), 2020 Fall 4

How many pixels? • iPhone 11 • 12MP (Mega Pixels) (approx. 4000 x 3000) • A virtual 45 Giga pixels camera http://360gigapixels.com/tokyo-tower-panorama-photo/ Visual Media Processing II (4098), 2020 Fall 5

Each pixel has RGB values • Red, Green, Blue R: 189 G: 207 B: 207 ja.wikipedia.org Visual Media Processing II (4098), 2020 Fall 6

Bit depth of image • 8bit (or 24bit) image • Each color channel value ranges between 0-255. • Typical images (e.g., JPEG) are 8bit depth. • Sufficient for normal use, insufficient for editing. • 16bit image • Each channel ranges 0-65535. • Contains more rich information. • Typical RAW data are 16bit depth. • More high bit / float value • For internal or scientific purpose. Xviii 18bit camera Visual Media Processing II (4098), 2020 Fall 7

Format of digital image • RAW image • Camera’s sensor data • 12MP x 3 channels x 2 bytes = 72 MB • Compressed image 圧縮 • Lossy. JPEG, GIF, HIEC, etc. Original • Values are grouped and approximated. 非可逆 10 times 100 times 1000 times • Lossless. PNG, TIF, etc. 可逆 • Preserves exactly same values. Some formats support 16bit. Visual Media Processing II (4098), 2020 Fall 8

Mini-report http://omilab.naist.jp/class/VMP2/2020/ • Q1. Check if explanation is true. A) For consumer purpose, lossy compression is widely used. B) For photometric analysis, lossless format is preferred. C) The storage size of lossless and RAW files are the same. D) The digital image has continuous values. E) Digital image is the array of integers spans among width, height, and channel. F) If you plan to edit the photo, camera should save 8bit image rather than 16bit. Visual Media Processing II (4098), 2020 Fall 9

Sensor • Array of buckets. Canon 6D sensor バケツ • Buckets collect photons. 光子 photons 光子 array of photon buckets close-up view of photon buckets To intensity 101 122 80 • Bare sensor does not distinguish the color. むき出しのセンサ Visual Media Processing II (4098), 2020 Fall 10

Pixel design helps photodiode collect more light (also called lenslet) microlens microlens color filter color filter photodiode photodiode フォトダイオード potential potential ポテンシャル井戸 well well silicon for read- stores emitted out etc. circuitry electrons made of silicon, emits 電子をためる 読出し回路などのシリコン層 electrons from photons 光子を電子に変換する Visual Media Processing II (4098), 2020 Fall 11

How to capture image? • Let’s say we have a sensor and an object to be photographed. real-world digital object sensor Visual Media Processing II (4098), 2020 Fall 12

How to capture image? • Lens map bundles of rays from scene point to the sensor pixel. Real-world object Lens Digital sensor Visual Media Processing II (4098), 2020 Fall 13

Mini-report http://omilab.naist.jp/class/VMP2/2020/ • Q2. Check if explanation is true. A) Sensor has bunch of photodiodes that collect photons. B) Sensor distinguishes spectral difference of light. C) Sensor distinguishes angular difference of rays. D) Camera lens map scene point to sensor pixel. E) A scene point contributes to all pixels if there is no lens. Visual Media Processing II (4098), 2020 Fall 14

Focal length 焦点距離 • Distance where parallel rays intersect. 𝑔 = 1 1 𝑏 + 1 𝑐 𝑐 𝑏 𝑔 • Fixed 𝑏 comparison 𝑏 Short focal length Moderate focal length Long focal length • Focal length determines field of view. 視野角 𝑔 𝑔 𝑔 Wide angle telephoto sensor sensor sensor Short focal length Moderate focal length Long focal length Visual Media Processing II (4098), 2020 Fall 15

Focal length and compression effect 圧縮効果 far near Field of view Wide Normal Narrow Focal length Small Moderate Large Perspective effect is lost. photo-studio9.com/compression-effect/ psy.ritsumei.ac.jp/ Visual Media Processing II (4098), 2020 Fall 16

Perspective distortion • Appearance is different long short Visual Media Processing II (4098), 2020 Fall 17

Focus and Depth of Field ピント 被写界深度 • Focus is not focal length. Focusing ( ピント合わせ ) • Effect of two (or more) lenses. Pooh 1 Pooh 2 In focus region Rilakuma ピントの合う位置 • Depth of field (DOF). (How deep light is focused) In focus region Pooh 1 Pooh 2 Rilakuma Wide DOF Shallow DOF Depth of field Visual Media Processing II (4098), 2020 Fall 18

Mini-report http://omilab.naist.jp/class/VMP2/2020/ • Q3. Check if explanation is true. A) A far object in a image is blurred because focal length was small. B) A far object in a image becomes small if focal length is small. C) Wide angle lens has small focal length. D) Wide DOF lens has small focal length. E) Focal length changes the appearance of the object. Visual Media Processing II (4098), 2020 Fall 20

Over/under exposure 露光 in highlights we are limited by clipping in shadows we are limited by noise Visual Media Processing II (4098), 2020 Fall

It’s very hard to capture all range at once • Digital image (or sensor) is linear scale. • World’s brightness is log scale. 明るさ 1 1500 25,000 400,000 2,000,000,000 Visual Media Processing II (4098), 2020 Fall

The world has a high dynamic range ダイナミックレンジ 10 -6 10 6 common real-world scenes adaptation range of our eyes Visual Media Processing II (4098), 2020 Fall

Digital sensors have a low dynamic range 10 -6 10 6 sensor 10 -6 10 6 common real-world scenes adaptation range of our eyes Visual Media Processing II (4098), 2020 Fall

(Digital) images have an even lower dynamic range low exposure 10 -6 10 6 image 10 -6 10 6 common real-world scenes adaptation range of our eyes Visual Media Processing II (4098), 2020 Fall

(Digital) images have an even lower dynamic range high exposure 10 -6 10 6 image 10 -6 10 6 common real-world scenes adaptation range of our eyes Visual Media Processing II (4098), 2020 Fall

Visual Media Processing II (4098), 2020 Fall

Visual Media Processing II (4098), 2020 Fall

Visual Media Processing II (4098), 2020 Fall

Visual Media Processing II (4098), 2020 Fall

High Dynamic Range (HDR) Fusion 合成 • Restore log scale luminance and map LDR values. 明るさ HDR Data Log scale luminance Fusion image Ordinary (LDR) images. HDR mode in iPhone camera. Visual Media Processing II (4098), 2020 Fall

Mini-report http://omilab.naist.jp/class/VMP2/2020/ • Q4 Check if explanation is true. A) Digital sensor’s dynamic range is wider than human eyes. B) Saturated region is black due to lack of photons. C) Sunny outdoor is approximately 10 times brighter than room light. D) HDR mode is suitable for capturing very dark and very bright region at the same time. Visual Media Processing II (4098), 2020 Fall 32

Three parameters to control brightness • Manual Mode Visual Media Processing II (4098), 2020 Fall

Aperture (Iris) 開口（絞り） • How many light rays are gathered. Small = dark Large = bright • Large aperture creates large bokeh. ボケ Single Pixel Large aperture = reach to neighbor pixels = out-of-focus Lens Small aperture = in-focus Aperture Visual Media Processing II (4098), 2020 Fall 34 センサ面

Aperture and DOF Nice bokeh! 1s, F14 = Wide DOF 1/30s, F2.2 = Shallow DOF Visual Media Processing II (4098), 2020 Fall 35

Shutter (Exposure time) シャッター速度（露光時間） • How long the sensor accumulate photons. • Long exposure suffers from motion blur. 動きブレ、手ブレ time time time Fast motion can be captured Splash is blurred Nikon-image.com Visual Media Processing II (4098), 2020 Fall 36

ISO (Sensitivity, Amplification ratio) ISO 感度（感度、増強率） • How to map electron levels to digital signal. • Noise is also amplified. x10 High ISO Equivalent long shutter Visual Media Processing II (4098), 2020 Fall 37

Best parameters depend on scene • Which effect do you want? Visual Media Processing II (4098), 2020 Fall

Mini-report http://omilab.naist.jp/class/VMP2/2020/ • Q5. Do parameters explain the photo? A) B) slow shutter, small aperture, low ISO fast shutter, large aperture, low ISO C) D) fast shutter, large aperture, high ISO fast shutter, small aperture, high ISO Visual Media Processing II (4098), 2020 Fall 39