Lecture 6: Math Review II Justin Johnson EECS 442 WI 2020: Lecture - PowerPoint PPT Presentation

Lecture 6: Math Review II Justin Johnson EECS 442 WI 2020: Lecture 6 - 1 January 28, 2020

Administrative • HW0 due tomorrow , 1/29 11:59pm • HW1 due 1 week from tomorrow , 2/5 11:59pm Justin Johnson EECS 442 WI 2020: Lecture 6 - 2 January 28, 2020

Last Time: Floating Point Math IEEE 754 Single Precision / Single / float32 8 bits 23 bits 2 127 ≈ 10 38 ≈ 7 decimal digits S Exponent Fraction IEEE 754 Double Precision / Double / float64 11 bits 52 bits 2 1023 ≈ 10 308 ≈ 15 decimal digits Exponent Fraction S Justin Johnson EECS 442 WI 2020: Lecture 6 - 3 January 28, 2020

Last Time: Vectors • Scale (vector, scalar → vector) • Add (vector, vector → vector) • Magnitude (vector → scalar) • Dot product (vector, vector → scalar) • Dot products are projection / angles • Cross product (vector, vector → vector) • Vectors facing same direction have cross product 0 • You can never mix vectors of different sizes Justin Johnson EECS 442 WI 2020: Lecture 6 - 4 January 28, 2020

Matrices Justin Johnson EECS 442 WI 2020: Lecture 6 - 5 January 28, 2020

Matrices Horizontally concatenate n, m-dim column vectors and you get a mxn matrix A (here 2x3) 𝑤 $ ) 𝑤 * ) 𝑤 + ) 𝑩 = 𝒘 $ , ⋯ , 𝒘 ' = 𝑤 $ , 𝑤 * , 𝑤 + , a undecorated a A (scalar) (vector) (matrix) lowercase lowercase uppercase bold or arrow bold Justin Johnson EECS 442 WI 2020: Lecture 6 - 6 January 28, 2020

Matrices Horizontally concatenate n, m-dim column vectors and you get a mxn matrix A (here 2x3) 𝑤 $ ) 𝑤 * ) 𝑤 + ) 𝑩 = 𝒘 $ , ⋯ , 𝒘 ' = 𝑤 $ , 𝑤 * , 𝑤 + , Watch out : In math, it’s common to treat D-dim vector as a Dx1 matrix (column vector); In numpy these are different things Justin Johnson EECS 442 WI 2020: Lecture 6 - 7 January 28, 2020

Matrices 𝑏 / Transpose: flip (3x1) T = 1x3 𝑐 = 𝑏 𝑐 𝑑 rows / columns 𝑑 Vertically concatenate m, n-dim row vectors and you get a mxn matrix A (here 2x3) / 𝒗 $ 𝑣 $ ) 𝑣 $ , 𝑣 $ 2 𝐵 = = ⋮ 𝑣 * ) 𝑣 * , 𝑣 * 2 / 𝒗 ' Justin Johnson EECS 442 WI 2020: Lecture 6 - 8 January 28, 2020

Matrix-vector Product 𝒛 *7$ = 𝑩 *7+ 𝒚 +7$ 𝑦 $ 𝑧 $ 𝒘 𝟐 𝒘 𝟑 𝒘 𝟒 𝑦 * 𝑧 * = 𝑦 + 𝒛 = 𝑦 $ 𝒘 𝟐 + 𝑦 * 𝒘 𝟑 + 𝑦 + 𝒘 𝟒 Linear combination of columns of A Justin Johnson EECS 442 WI 2020: Lecture 6 - 9 January 28, 2020

Matrix-vector Product 𝒛 *7$ = 𝑩 *7+ 𝒚 +7$ 3 𝑧 $ 𝑼 𝒗 𝟐 𝑧 * = 𝒚 3 𝑼 𝒗 𝟑 𝑼 𝒚 𝑼 𝒚 𝑧 $ = 𝒗 𝟐 𝑧 * = 𝒗 𝟑 Dot product between rows of A and x Justin Johnson EECS 442 WI 2020: Lecture 6 - 10 January 28, 2020

Matrix Multiplication Generally: A mn and B np yield product ( AB ) mp | | 𝑼 − 𝒃 𝟐 − 𝒄 𝟐 ⋯ 𝒄 𝒒 𝑩𝑪 = ⋮ 𝑼 − 𝒃 𝒏 − | | Yes – in A , I’m referring to the rows, and in B , I’m referring to the columns Justin Johnson EECS 442 WI 2020: Lecture 6 - 11 January 28, 2020

Matrix Multiplication Generally: A mn and B np yield product ( AB ) mp | | 𝑼 𝒄 𝒌 𝑩𝑪 HI = 𝒃 𝒋 𝒄 𝟐 ⋯ 𝒄 𝒒 | | 𝑼 𝒄 𝟐 𝑼 𝒄 𝒒 𝑼 𝒃 𝟐 ⋯ 𝒃 𝟐 − 𝒃 𝟐 − 𝑩𝑪 = ⋮ ⋮ ⋱ ⋮ 𝑼 𝒄 𝟐 𝑼 𝒄 𝒒 𝑼 − 𝒃 𝒏 − 𝒃 𝒏 ⋯ 𝒃 𝒏 Justin Johnson EECS 442 WI 2020: Lecture 6 - 12 January 28, 2020

Matrix Multiplication • Dimensions must match • Dimensions must match • Dimensions must match • (Associative): ABx = (A)(Bx) = (AB)x • (Not Commutative): ABx ≠ (BA)x ≠ (BxA) Justin Johnson EECS 442 WI 2020: Lecture 6 - 13 January 28, 2020

Two uses for Matrices 1. Storing things in a rectangular array (e.g. images) • Typical operations : element-wise operations, convolution (which we’ll cover later) • Atypical operations : almost anything you learned in a math linear algebra class 2. A linear operator that maps vectors to another space ( Ax ) • Typical/Atypical: reverse of above Justin Johnson EECS 442 WI 2020: Lecture 6 - 14 January 28, 2020

Images as Matrices Suppose someone hands you this matrix. What’s wrong with it? No contrast! Justin Johnson EECS 442 WI 2020: Lecture 6 - 15 January 28, 2020

Contrast: Gamma Curve Typical way to change the contrast is to apply a nonlinear correction pixelvalue T The quantity 𝛿 controls how much contrast gets added Justin Johnson EECS 442 WI 2020: Lecture 6 - 16 January 28, 2020

Contrast: Gamma Curve Now the darkest 90% regions (10 th pctile) are much darker than the 50% moderately dark regions (50 th pctile). new 10% 90% new new 10% 50% Justin Johnson EECS 442 WI 2020: Lecture 6 - 17 January 28, 2020

Contrast: Gamma Correction Justin Johnson EECS 442 WI 2020: Lecture 6 - 18 January 28, 2020

Contrast: Gamma Correction Phew! Much Better. Justin Johnson EECS 442 WI 2020: Lecture 6 - 19 January 28, 2020

Implementation Python+Numpy (right way): imNew = im**4 Python+Numpy (slow way – why? ): imNew = np.zeros(im.shape) for y in range(im.shape[0]): for x in range(im.shape[1]): imNew[y,x] = im[y,x]**expFactor Justin Johnson EECS 442 WI 2020: Lecture 6 - 20 January 28, 2020

Elementwise Operations Element-wise power – beware notation Z 𝑩 Z HI = 𝐵 HI “Hadamard Product” / Element-wise multiplication 𝑩 ⊙ 𝑪 HI = 𝑩 HI ∗ 𝑪 HI Element-wise division 𝑩/𝑪 HI = 𝐵 HI 𝐶 HI Justin Johnson EECS 442 WI 2020: Lecture 6 - 21 January 28, 2020

Sums Across Axes 𝑦 $ 𝑧 $ Suppose have ⋮ ⋮ 𝑩 = Nx2 matrix A 𝑦 ' 𝑧 ' 𝑦 $ + 𝑧 $ ⋮ Σ(𝑩, 1) = ND col. vec. 𝑦 ' + 𝑧 ' ' ' Σ(𝑩, 0) = ` 𝑦 H , ` 𝑧 H 2D row vec Ha$ Ha$ Note – libraries distinguish between N-D column vector and Nx1 matrix. Justin Johnson EECS 442 WI 2020: Lecture 6 - 22 January 28, 2020

Operations they don’t teach You Probably Saw Matrix Addition 𝑒 + 𝑓 𝑔 𝑏 + 𝑓 𝑐 + 𝑔 𝑏 𝑐 ℎ = 𝑕 𝑑 + 𝑕 𝑒 + ℎ 𝑑 What is this? FYI: e is a scalar 𝑏 + 𝑓 𝑐 + 𝑓 𝑏 𝑐 𝑒 + 𝑓 = 𝑑 𝑑 + 𝑓 𝑒 + 𝑓 Justin Johnson EECS 442 WI 2020: Lecture 6 - 23 January 28, 2020

Broadcasting If you want to be pedantic and proper, you expand e by multiplying a matrix of 1s (denoted 1 ) 𝑏 𝑐 = 𝑏 𝑐 𝑒 + 𝑓 𝑒 + 𝟐 *7* 𝑓 𝑑 𝑑 𝑒 + 𝑓 𝑓 = 𝑏 𝑐 𝑓 𝑓 𝑑 Many smart matrix libraries do this automatically. This is the source of many bugs. Justin Johnson EECS 442 WI 2020: Lecture 6 - 24 January 28, 2020

Broadcasting Example Given: a nx2 matrix P and a 2D column vector v , Want: nx2 difference matrix D 𝑦 $ 𝑧 $ 𝑦 $ − 𝑏 𝑧 $ − 𝑐 𝒘 = 𝑏 ⋮ ⋮ 𝑸 = ⋮ ⋮ 𝑬 = 𝑐 𝑦 ' 𝑧 ' 𝑦 ' − 𝑏 𝑧 ' − 𝑐 𝑦 $ 𝑧 $ Blue stuff is 𝑏 𝑐 𝑸 − 𝒘 / = ⋮ ⋮ − ⋮ assumed / 𝑦 ' 𝑧 ' 𝑏 𝑐 broadcast Justin Johnson EECS 442 WI 2020: Lecture 6 - 25 January 28, 2020

Broadcasting Rules Suppose we have numpy arrays x and y. How will they broadcast? 1. Write down the shape of each array as a tuple of integers: For example: x: (10,) y: (20, 10) 2. If they have different numbers of dimensions, prepend with ones until they have the same number of dimensions For example: x: (10,) y: (20, 10) à x: (1, 10) y: (20, 10) 3. Compare each dimension. There are 3 cases: (a) Dimension match. Everything is good (b) Dimensions don’t match, but one is =1. ”Duplicate” the smaller array along that axis to match (c) Dimensions don’t match, neither are =1. Error! Justin Johnson EECS 442 WI 2020: Lecture 6 - 26 January 28, 2020

Broadcasting Examples x = np.ones(10, 20) x = np.ones(10, 20) y = np.ones(20) y = np.ones(10) z = x + y z = x + y print(z.shape) print(z.shape) (10,20) ERROR x = np.ones(10, 20) x = np.ones(1, 20) y = np.ones(10, 1) y = np.ones(10, 1) z = x + y z = x + y print(z.shape) print(z.shape) (10,20) (10,20) Justin Johnson EECS 442 WI 2020: Lecture 6 - 27 January 28, 2020

Tensors Scalar : Just one number Vector : 1D list of numbers Matrix : 2D grid of numbers Tensor : N-dimensional grid of numbers (Lots of other meanings in math, physics) Justin Johnson EECS 442 WI 2020: Lecture 6 - 28 January 28, 2020

Broadcasting with Tensors The same broadcasting rules apply to tensors with any number of dimensions! x = np.ones(30) y = np.ones(20, 1) z = np.ones(10, 1, 1) w = x + y + z print(w.shape) (10, 20, 30) Justin Johnson EECS 442 WI 2020: Lecture 6 - 29 January 28, 2020

Vectorization Writing code without explicit loops: use broadcasting, matrix multiply, and other (optimized) numpy primitives instead Justin Johnson EECS 442 WI 2020: Lecture 6 - 30 January 28, 2020

Vectorization Example • Suppose I represent each image as a 128- dimensional vector • I want to compute all the pairwise distances between { x 1 , …, x N } and { y 1 , …, y M } so I can find, for every x i the nearest y j • Identity: 𝒚 − 𝒛 * = 𝒚 * + 𝒛 * − 2𝒚 / 𝒛 𝒚 * + 𝒛 * − 2𝒚 / 𝒛 $/* • Or: 𝒚 − 𝒛 = Justin Johnson EECS 442 WI 2020: Lecture 6 - 31 January 28, 2020