Image Processing Todays Class Image Representations: Matrices - PowerPoint PPT Presentation

CS6501: Deep Learning for Visual Recognition Image Processing

Today’s Class Image Representations: Matrices Image Representations: RGB, HSV, etc Image Processing: Brightness Image Filtering: Mean Filter, Median Filter Image Filtering: Convolutions Blurring, Sharpening, Computing Gradients Walkthrough Image Processing Primer

About the Course CS6501-003: Deep Learning for Visual Recognition • Instructor: Vicente Ordóñez • Email: vicente@virginia.edu • Website: http://vicenteordonez.com/deeplearning/ • Class Location: Olsson Hall 005 • Class Times: Monday-Wednesday 3:30pm and 4:45pm • Piazza: https://piazza.com/virginia/spring2020/cs6501003/home • Office Hours: Tuesdays 3 to 5pm (Rice 310) 3

Teaching Assistants Ziyan Yang Paola Cascante-Bonilla (tw8cb@virginia.edu) (pc9za@virginia.edu) Office Hours: Thursdays Hours: Fridays 2 to 4pm 3 to 5pm (Rice 442) (Rice 442) 4

Grading • Assignments: 400pts (4 assignments) (100pts + 100pts + 100pts + 100pts) • Course Project: 400pts Groups of up to 3 students (more only if justified) • Paper Reading Summaries: 100pts • Class Paper Presentation: 100pts (groups of mostly 2 students) 5

Also… • Assignment 1 released on course website • You should have completed the pytorch/jupyter/Google Colaboratory tutorial and the Numpy / Image Processing tutorial.

Additionally you will need for your Course Project: g2.2xlarge: 4GB ($0.65 / hour) p2.xlarge: 12GB ($0.90 / hour) Free credits for students! $50.

Or maybe, even better: GTX 1080 Ti (11GB): $700 GTX 1080 (8GB): $500 GTX 1060 (6GB): $150-$250

I am also working to get you set up on GTX 1080 Ti (11GB): $700 GTX 1080 (8GB): $500 GTX 1060 (6GB): $150-$250

Reminder of what is an image for a computer. What we see What a computer sees Source: S. Narasimhan

Images as Functions z = #(%, ')

Images as Functions z = #(%, ') • The domain of x and y is [0, img-width) and [0 and img-height) x, and y are discretized into integer values. •

Light • What determines the color of a pixel? Figure from Szeliski

The Retina Cross-section of eye Cross section of retina Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision Pigmented epithelium Ganglion axons Ganglion cell layer Bipolar cell layer [What the Frog's Eye Tells the Frog's Brain] Receptor layer

Electromagnetic Spectrum Human Luminance Sensitivity Function http://www.yorku.ca/eye/photopik.htm

Basic Image Processing ! "! " > 1

Basic Image Processing ! "! 0 < " < 1

Color Images as Tensors !ℎ#$$%& ' ℎ%()ℎ* ' +(,*ℎ

Color Images as Tensors !ℎ#$$%& ' ℎ%()ℎ* ' +(,*ℎ Channels are usually RGB: Red, Green, and Blue Other color spaces: HSV, HSL, LUV, XYZ, Lab, CMYK, etc

Color spaces: RGB 0,1,0 R (G=0,B=0) G 1,0,0 (R=0,B=0) 0,0,1 B Some drawbacks (R=0,G=0) • Strongly correlated channels • Non-perceptual Default color space Slide by James Hays Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png

Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0) Slide by James Hays

Color spaces: L*a*b* “Perceptually uniform”* color space L (a=0,b=0) a (L=65,b=0) b (L=65,a=0) Slide by James Hays

Most information in intensity Only color shown – constant intensity Slide by James Hays

Most information in intensity Only intensity shown – constant color Slide by James Hays

Most information in intensity Original image Slide by James Hays

Image filtering

Image filtering Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

Image filtering Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

Image filtering: e.g. Mean Filter

Image filtering: e.g. Mean Filter

Image filtering: e.g. Median Filter Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

Image filtering: Convolution operator !(#, %) ' #, % = ) ) ! ,, - .(# − ,, % − -) * + Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

(filter, kernel) http://www.cs.virginia.edu/~vicente/recognition/animation.gif

Image filtering: Convolution operator e.g. mean filter 1/9 1/9 1/9 1/9 1/9 1/9 !(#, %) = 1/9 1/9 1/9 !(#, %) Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

Image filtering: Convolution operator e.g. mean filter 1/9 1/9 1/9 1/9 1/9 1/9 !(#, %) = 1/9 1/9 1/9 !(#, %) Image Credit: http://what-when-how.com/introduction-to-video-and-image-processing/neighborhood-processing-introduction-to-video-and-image-processing-part-1/

Example: box filter × × g [ , ] 1 1 1 1 1 1 1 1 1 Slide credit: David Lowe (UBC)

Image filtering 1 1 1 × × g [ , ] 1 1 1 1 1 1 h [.,.] f [.,.] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = å + + h [ m , n ] g [ k , l ] f [ m k , n l ] k , l Credit: S. Seitz

Image filtering 1 1 1 × × g [ , ] 1 1 1 1 1 1 h [.,.] f [.,.] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 0 0 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = å + + h [ m , n ] g [ k , l ] f [ m k , n l ] k , l Credit: S. Seitz

Image filtering 1 1 1 × × g [ , ] 1 1 1 1 1 1 h [.,.] f [.,.] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 0 0 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = å + + h [ m , n ] g [ k , l ] f [ m k , n l ] k , l Credit: S. Seitz

Image filtering 1 1 1 × × g [ , ] 1 1 1 1 1 1 h [.,.] f [.,.] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 0 0 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = å + + h [ m , n ] g [ k , l ] f [ m k , n l ] k , l Credit: S. Seitz

Image filtering 1 1 1 × × g [ , ] 1 1 1 1 1 1 h [.,.] f [.,.] 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = å + + h [ m , n ] g [ k , l ] f [ m k , n l ] k , l Credit: S. Seitz