Lecture 7: More Math + Image Filtering Justin Johnson EECS 442 WI - PowerPoint PPT Presentation

Lecture 7: More Math + Image Filtering Justin Johnson EECS 442 WI 2020: Lecture 7 - 1 January 30, 2020

Administrative HW0 was due yesterday! HW1 due a week from yesterday Justin Johnson EECS 442 WI 2020: Lecture 7 - 2 January 30, 2020

Cool Talk Today: https://cse.engin.umich.edu/event/numpy-a-look-at-the-past-present-and-future-of-array-computation Justin Johnson EECS 442 WI 2020: Lecture 7 - 3 January 30, 2020

Last Time: Matrices, Vectorization, Linear Algebra Justin Johnson EECS 442 WI 2020: Lecture 7 - 4 January 30, 2020

Eigensystems • An eigenvector 𝒘 𝒋 and eigenvalue 𝜇 $ of a matrix 𝑩 satisfy 𝑩𝒘 𝒋 = 𝜇 $ 𝒘 𝒋 ( 𝑩𝒘 𝒋 is scaled by 𝜇 $ ) • Vectors and values are always paired and typically ' = 1 you assume 𝒘 𝒋 • Biggest eigenvalue of A gives bounds on how much 𝑔 𝒚 = 𝑩𝒚 stretches a vector x . • Hints of what people really mean: • “Largest eigenvector” = vector w/ largest value • “Spectral” just means there’s eigenvectors somewhere Justin Johnson EECS 442 WI 2020: Lecture 7 - 5 January 30, 2020

Suppose I have points in a grid Justin Johnson EECS 442 WI 2020: Lecture 7 - 6 January 30, 2020

Now I apply f( x ) = Ax to these points Pointy-end: Ax . Non-Pointy-End: x Justin Johnson EECS 442 WI 2020: Lecture 7 - 7 January 30, 2020

𝑩 = 1.1 0 0 1.1 Red box – unit square, Blue box – after f( x ) = Ax . What are the yellow lines and why? Justin Johnson EECS 442 WI 2020: Lecture 7 - 8 January 30, 2020

𝑩 = 0.8 0 0 1.25 Now I apply f( x ) = Ax to these points Pointy-end: Ax . Non-Pointy-End: x Justin Johnson EECS 442 WI 2020: Lecture 7 - 9 January 30, 2020

𝑩 = 0.8 0 0 1.25 Red box – unit square, Blue box – after f( x ) = Ax . What are the yellow lines and why? Justin Johnson EECS 442 WI 2020: Lecture 7 - 10 January 30, 2020

𝑩 = cos(𝑢) −sin(𝑢) sin(𝑢) cos(𝑢) Red box – unit square, Blue box – after f( x ) = Ax . Can we draw any yellow lines? Justin Johnson EECS 442 WI 2020: Lecture 7 - 11 January 30, 2020

Eigenvectors of Symmetric Matrices • Always n mutually orthogonal eigenvectors with n (not necessarily) distinct eigenvalues • For symmetric 𝑩 , the eigenvector with the largest 𝒚 𝑼 𝑩𝒚 eigenvalue maximizes 𝒚 𝑼 𝒚 (smallest/min) • So for unit vectors (where 𝒚 𝑼 𝒚 = 1 ), that eigenvector maximizes 𝒚 𝑼 𝑩𝒚 • A surprisingly large number of optimization problems rely on (max/min)imizing this Justin Johnson EECS 442 WI 2020: Lecture 7 - 12 January 30, 2020

Singular Value Decomposition Can always write a mxn matrix A as: 𝑩 = 𝑽𝚻𝑾 𝑼 0 σ 1 A = U ∑ σ 2 σ 3 0 Scale Rotation Eigenvectors Sqrt of of AA T Eigenvalues of A T A Justin Johnson EECS 442 WI 2020: Lecture 7 - 13 January 30, 2020

Singular Value Decomposition Can always write a mxn matrix A as: 𝑩 = 𝑽𝚻𝑾 𝑼 V T A = U ∑ Rotation Scale Rotation Eigenvectors Sqrt of Eigenvectors of AA T Eigenvalues of A T A of A T A Justin Johnson EECS 442 WI 2020: Lecture 7 - 14 January 30, 2020

Singular Value Decomposition • Every matrix is a rotation, scaling, and rotation • Number of non-zero singular values = rank / number of linearly independent vectors • “Closest” matrix to A with a lower rank 0 σ 1 V T σ 2 = A U σ 3 0 Justin Johnson EECS 442 WI 2020: Lecture 7 - 15 January 30, 2020

Singular Value Decomposition • Every matrix is a rotation, scaling, and rotation • Number of non-zero singular values = rank / number of linearly independent vectors • “Closest” matrix to A with a lower rank 0 σ 1 V T σ 2 = Â U 0 0 Justin Johnson EECS 442 WI 2020: Lecture 7 - 16 January 30, 2020

Singular Value Decomposition • Every matrix is a rotation, scaling, and rotation • Number of non-zero singular values = rank / number of linearly independent vectors • “Closest” matrix to A with a lower rank • Secretly behind basically many things you do with matrices Justin Johnson EECS 442 WI 2020: Lecture 7 - 17 January 30, 2020

Solving Least-Squares Start with two points (x i ,y i ) (x 2 ,y 2 ) 𝒛 = 𝑩𝒘 𝑧 @ 𝑧 ' = 𝑦 @ 1 𝑛 𝑦 ' 1 𝑐 (x 1 ,y 1 ) 𝑧 @ 𝑧 ' = 𝑛𝑦 @ + 𝑐 𝑛𝑦 ' + 𝑐 We know how to solve this – invert A and find v (i.e., (m,b) that fits points) Justin Johnson EECS 442 WI 2020: Lecture 7 - 18 January 30, 2020

Solving Least-Squares Start with two points (x i ,y i ) (x 2 ,y 2 ) 𝒛 = 𝑩𝒘 𝑧 @ 𝑧 ' = 𝑦 @ 1 𝑛 𝑦 ' 1 𝑐 (x 1 ,y 1 ) ' 𝑧 @ 𝑧 ' − 𝑛𝑦 @ + 𝑐 𝒛 − 𝑩𝒘 ' = 𝑛𝑦 ' + 𝑐 ' + 𝑧 ' − 𝑛𝑦 ' + 𝑐 ' = 𝑧 @ − 𝑛𝑦 @ + 𝑐 The sum of squared differences between the actual value of y and what the model says y should be. Justin Johnson EECS 442 WI 2020: Lecture 7 - 19 January 30, 2020

Solving Least-Squares Suppose there are n > 2 points 𝒛 = 𝑩𝒘 𝑧 @ 𝑦 @ 1 𝑛 ⋮ ⋮ ⋮ = 𝑐 𝑧 G 𝑦 G 1 Compute 𝑧 − 𝐵𝑦 ' again K 𝒛 − 𝑩𝒘 ' = I 𝑧 $ − (𝑛𝑦 $ + 𝑐) ' $J@ Justin Johnson EECS 442 WI 2020: Lecture 7 - 20 January 30, 2020

Solving Least-Squares Given y , A , and v with y = Av overdetermined ( A tall / more equations than unknowns) We want to minimize 𝒛 − 𝑩𝒘 𝟑 , or find: arg min 𝒘 𝒛 − 𝑩𝒘 ' (The value of x that makes the expression smallest) Solution satisfies 𝑩 𝑼 𝑩 𝒘 ∗ = 𝑩 𝑼 𝒛 or R@ 𝑩 𝑼 𝒛 𝒘 ∗ = 𝑩 𝑼 𝑩 (Don’t actually compute the inverse!) Justin Johnson EECS 442 WI 2020: Lecture 7 - 21 January 30, 2020

When is Least-Squares Possible? Given y , A , and v . Want y = Av Want n outputs, have n knobs to y = A v fiddle with, every knob is useful if A is full rank. A: rows (outputs) > columns = v (knobs). Thus can’t get precise y A output you want (not enough knobs). So settle for “closest” knob setting. Justin Johnson EECS 442 WI 2020: Lecture 7 - 22 January 30, 2020

When is Least-Squares Possible? Given y , A , and v . Want y = Av Want n outputs, have n knobs to y = A v fiddle with, every knob is useful if A is full rank. A: columns (knobs) > rows y = A v (outputs). Thus, any output can be expressed in infinite ways. Justin Johnson EECS 442 WI 2020: Lecture 7 - 23 January 30, 2020

Homogeneous Least-Squares Given a set of unit vectors (aka directions) 𝒚 𝟐 , … , 𝒚 𝒐 and I want vector 𝒘 that is as orthogonal to all the 𝒚 𝒋 as possible (for some definition of orthogonal) Stack 𝒚 𝒋 into A , compute Av 𝒚 𝒐 … 𝒚 𝟑 0 if 𝑼 𝑼 𝒘 − 𝒚 𝟐 − 𝒚 𝟐 𝒚 𝟐 orthog 𝑩𝒘 = 𝒘 = ⋮ ⋮ 𝑼 𝑼 𝒘 − 𝒚 𝒐 − 𝒚 𝒐 𝒐 𝒘 𝟑 𝑩𝒘 𝟑 = I 𝑼 𝒘 Compute 𝒚 𝒋 𝒋 Sum of how orthog. v is to each x Justin Johnson EECS 442 WI 2020: Lecture 7 - 24 January 30, 2020

Homogenous Least-Squares • A lot of times, given a matrix A we want to find the v that minimizes 𝑩𝒘 ' . • I.e., want 𝐰 ∗ = arg min ' 𝑩𝒘 ' 𝒘 • What’s a trivial solution? • Set v = 0 → Av = 0 • Exclude this by forcing v to have unit norm Justin Johnson EECS 442 WI 2020: Lecture 7 - 25 January 30, 2020

Homogenous Least-Squares ' Let’s look at 𝑩𝒘 ' ' = 𝑩𝒘 ' Rewrite as dot product ' = 𝐁𝐰 Z (𝐁𝐰) 𝑩𝒘 ' Distribute transpose ' = 𝒘 𝑼 𝑩 𝑼 𝐁𝐰 = 𝐰 𝐔 𝐁 𝐔 𝐁 𝐰 𝑩𝒘 ' We want the vector minimizing this quadratic form Where have we seen this? Justin Johnson EECS 442 WI 2020: Lecture 7 - 26 January 30, 2020

Homogenous Least-Squares Ubiquitious tool in vision: 𝒘 [ J@ 𝑩𝒘 ' arg min (1) “Smallest”* eigenvector of 𝑩 𝑼 𝑩 (2) “ Smallest” right singular vector of 𝑩 For min → max, switch smallest → largest *Note: 𝑩 𝑼 𝑩 is positive semi-definite so it has all non-negative eigenvalues Justin Johnson EECS 442 WI 2020: Lecture 7 - 27 January 30, 2020

Derivatives Justin Johnson EECS 442 WI 2020: Lecture 7 - 28 January 30, 2020

Derivatives Remember derivatives? Derivative: rate at which a function f(x) changes at a point as well as the direction that increases the function Justin Johnson EECS 442 WI 2020: Lecture 7 - 29 January 30, 2020

Given quadratic function f(x) 𝑔 𝑦, 𝑧 = 𝑦 − 2 ' + 5 𝑔 𝑦 is function 𝑕 𝑦 = 𝑔 ] 𝑦 aka 𝑕 𝑦 = 𝑒 𝑒𝑦 𝑔(𝑦) Justin Johnson EECS 442 WI 2020: Lecture 7 - 30 January 30, 2020

Given quadratic function f(x) 𝑔 𝑦, 𝑧 = 𝑦 − 2 ' + 5 What’s special about x=2? 𝑔 𝑦 minim. at 2 𝑕 𝑦 = 0 at 2 a = minimum of f → 𝑕 𝑏 = 0 Reverse is not true Justin Johnson EECS 442 WI 2020: Lecture 7 - 31 January 30, 2020

Rates of change 𝑔 𝑦, 𝑧 = 𝑦 − 2 ' + 5 Suppose I want to increase f(x) by changing x: Blue area: move left Red area: move right Derivative tells you direction of ascent and rate Justin Johnson EECS 442 WI 2020: Lecture 7 - 32 January 30, 2020

Calculus to Know • Really need intuition • Need chain rule • Rest you should look up / use a computer algebra system / use a cookbook • Partial derivatives (and that’s it from multivariable calculus) Justin Johnson EECS 442 WI 2020: Lecture 7 - 33 January 30, 2020