Lecture 1: Review of DTFT, Gaussians, and Linear Algebra Mark - PowerPoint PPT Presentation

Outline DTFT Gaussians Linear Algebra Summary Lecture 1: Review of DTFT, Gaussians, and Linear Algebra Mark Hasegawa-Johnson ECE 417: Multimedia Signal Processing, Fall 2020

Outline DTFT Gaussians Linear Algebra Summary Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary Outline Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary Outline of today’s lecture 1 Syllabus 2 Homework 1 3 Review: DTFT, Gaussians, and Linear Algebra

Outline DTFT Gaussians Linear Algebra Summary What are the pre-requisites for ECE 417? ECE 310 Digital Signal Processing ECE 313 Probability with Engineering Applications Math 286 Intro to Differential Eq Plus

Outline DTFT Gaussians Linear Algebra Summary Outline Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary Discrete-Time Fourier Transform The discrete-time Fourier transform of a signal x [ n ] is ∞ � x [ n ] e − j ω n X ( ω ) = n = −∞ The inverse DTFT is � π x [ n ] = 1 X ( ω ) e j ω n d ω 2 π − π

Outline DTFT Gaussians Linear Algebra Summary DTFT of a rectangle One of the most important DTFTs you should know is the DTFT of a length- N rectangle: � 1 0 ≤ n ≤ N − 1 x [ n ] = u [ n ] − u [ n − N ] = 0 otherwise It is N − 1 e − j ω n = 1 − e − j ω N 2 ) sin( ω N / 2) 1 − e − j ω = e − j ω ( N − 1 � X ( ω ) = sin( ω/ 2) n =0

Outline DTFT Gaussians Linear Algebra Summary Smith, J.O. ”The Rectangular Window”, in Spectral Audio Signal Processing, online book , 2011 edition.

Outline DTFT Gaussians Linear Algebra Summary

Outline DTFT Gaussians Linear Algebra Summary Outline Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary Gaussian (a.k.a. normal) pdf By InductiveLoad, public domain, https://commons.wikimedia.org/wiki/File:Normal_Distribution_PDF.svg

Outline DTFT Gaussians Linear Algebra Summary Normal pdf A Gaussian random variable, X , is one whose probability density function is given by ( x − µ )2 1 2 πσ 2 e − 1 √ f X ( x ) = σ 2 2 where µ and σ 2 are the mean and variance, σ 2 = E ( X − µ ) 2 � � µ = E [ X ] ,

Outline DTFT Gaussians Linear Algebra Summary Standard normal The cumulative distribution function (CDF) of a Gaussian RV is � x � ( x − µ ) /σ � x − µ � F X ( x ) = P { X ≤ x } = f X ( y ) dy = f Z ( y ) dy = Φ σ −∞ −∞ where Z = X − µ is called the standard normal random variable. It σ is a Gaussian with zero mean, and unit variance: 1 e − 1 2 z 2 f Z ( z ) = √ 2 π We define Φ( z ) to be the CDF of the standard normal RV: � z Φ( z ) = f Z ( y ) dy −∞

Outline DTFT Gaussians Linear Algebra Summary Multivariate normal pdf By Bscan, public domain, https://commons.wikimedia.org/wiki/File:MultivariateNormal.png

Outline DTFT Gaussians Linear Algebra Summary Jointly Gaussian Random Variables Two random variables, X 1 and X 2 , are jointly Gaussian if 1 µ ) T Σ − 1 ( � 2 π | Σ | 1 / 2 e − 1 2 ( � x − � x − � µ ) f X 1 , X 2 ( x 1 , x 2 ) = where � X is the random vector, � µ is its mean, and Σ is its covariance matrix, � X 1 � � � � � � � ( � µ ) T ( � X = , � µ = E , Σ = E X − � X − � µ ) X X 2

Outline DTFT Gaussians Linear Algebra Summary Covariance The covariance matrix has four elements: � σ 2 � ρ 12 1 Σ = σ 2 ρ 21 2 σ 2 1 and σ 2 2 are the variances of X 1 and X 2 , respectively. ρ 12 = ρ 21 is the covariance of X 1 and X 2 : µ 1 = E [ X 1 ] σ 2 ( X 1 − µ 1 ) 2 � � 1 = E σ 2 ( X 2 − µ 2 ) 2 � � 2 = E ρ 12 = E [( X 1 − µ 1 )( X 2 − µ 2 )]

Outline DTFT Gaussians Linear Algebra Summary Jointly Gaussian Random Variables 1 µ ) T Σ − 1 ( � 2 π | Σ | 1 / 2 e − 1 2 ( � x − � x − � µ ) f X 1 , X 2 ( x 1 , x 2 ) = The multivariate normal pdf contains the determinant and the inverse of Σ. For a two-dimensional vector � X , these are � σ 2 � ρ 12 1 Σ = σ 2 ρ 21 2 | Σ | = σ 2 1 σ 2 2 − ρ 12 ρ 21 � σ 2 � Σ − 1 = 1 − ρ 12 2 σ 2 − ρ 21 | Σ | 1

Outline DTFT Gaussians Linear Algebra Summary Gaussian: Uncorrelated ⇔ Independent Notice that if two Gaussian random variables are uncorrelated ( ρ 12 = 0), then they are also independent: T   σ 2  x 1 − µ 1   0   x 1 − µ 1   2       σ 2 x 2 − µ 2 0 x 2 − µ 2    1 − 1 1 σ 2 1 σ 2 2 f X 1 , X 2 ( x 1 , x 2 ) = 2 π | Σ | 1 / 2 e 2 � ( x 1 − µ 1)2 + ( x 2 − µ 2)2 � − 1 1 σ 2 σ 2 2 = e 1 2 2 πσ 1 σ 2     � x 1 − µ 2 � x 2 − µ 2 � 2 � 2 1 1 e − 1 e − 1 = 2 2 σ 1 σ 2     � � 2 πσ 2 2 πσ 2 1 2 = f X 1 ( x 1 ) f X 2 ( x 2 )

Outline DTFT Gaussians Linear Algebra Summary Outline Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary A linear transform � y = A � x maps vector space � x onto vector space � y . For example: � 1 � 1 the matrix A = maps the vectors 0 2 � x 0 , � x 1 , � x 2 , � x 3 = � 1 � 0 � 1 � � − 1 � � � √ √ 2 2 , , , 1 1 0 1 √ √ 2 2 to the vectors � y 0 , � y 1 , � y 2 , � y 3 = � √ � 1 � 1 � � � � � 2 0 √ √ , , , 0 2 2 2

Outline DTFT Gaussians Linear Algebra Summary A linear transform � y = A � x maps vector space � x onto vector space � y . The absolute value of the determinant of A tells you how much the area of a unit circle is changed under the transformation. � 1 � 1 For example, if A = , then the 0 2 unit circle in � x (which has an area of π ) is mapped to an ellipse with an area that is abs( | A | ) = 2 times larger, i.e., i.e., π abs( | A | ) = 2 π .

Outline DTFT Gaussians Linear Algebra Summary For a D-dimensional square matrix, there may be up to D different directions � x = � v d such that, for some scalar λ d , A � v d = λ d � v d . � 1 � 1 For example, if A = , then the 0 2 eigenvectors are � 1 � 1 � � √ 2 v 0 = � , v 1 = � , 1 0 √ 2 and the eigenvalues are λ 0 = 1, λ 1 = 2. Those vectors are red and extra-thick, in the figure to the left. Notice that one of the vectors gets scaled by λ 0 = 1, but the other gets scaled by λ 1 = 2.

Outline DTFT Gaussians Linear Algebra Summary An eigenvector is a direction, not just a vector. That means that if you multiply an eigenvector by any scalar, you get the same eigenvector: if A � v d = λ d � v d , then its also true that cA � v d = c λ d � v d for any scalar c . For example: the following are the same eigenvector as � v 1 � 1 � − 1 � √ � √ 2 2 � v 1 = , − � v 1 = − 1 1 √ 2 Since scale and sign don’t matter, by convention, we normalize so that an eigenvector is always unit-length ( � � v d � = 1) and the first nonzero element is non-negative ( v d 0 > 0).

Outline DTFT Gaussians Linear Algebra Summary Eigenvalues: Before you find the eigenvectors, you should first find the eigenvalues. You can do that using this fact: A � v d = λ d � v d A � v d = λ d I � v d v d = � A � v d − λ d I � 0 v d = � ( A − λ d I ) � 0 That means that when you use the linear transform ( A − λ d I ) to transform the unit circle, the result has an area of | A − λ I | = 0.

Outline DTFT Gaussians Linear Algebra Summary Example: � � 1 − λ 1 � � | A − λ I | = � � 0 2 − λ � � = 2 − 3 λ + λ 2 which has roots at λ 0 = 1, λ 1 = 2

Outline DTFT Gaussians Linear Algebra Summary There are always D eigenvalues The determinant | A − λ I | is a D th -order polynomial in λ . By the fundamental theorem of algebra, the equation | A − λ I | = 0 has exactly D roots (counting repeated roots and complex roots). Therefore, any square matrix has exactly D eigenvalues (counting repeated eigenvalues, and complex eigenvalues. The same is not true of eigenvalues. Not every square matrix has eigenvectors . Complex and repeated eigenvalues usually correspond to eigensubspaces, not eigenvectors.

Outline DTFT Gaussians Linear Algebra Summary Outline Outline of today’s lecture 1 Review: DTFT 2 Review: Gaussians 3 Review: Linear Algebra 4 Summary 5

Outline DTFT Gaussians Linear Algebra Summary Summary DTFT of a rectangle: 2 ) sin( ω N / 2) x [ n ] = u [ n ] − u [ n − N ] ↔ X ( ω ) = e − j ω ( N − 1 sin( ω/ 2) Jointly Gaussian RVs: 1 µ ) T Σ − 1 ( � 2 π | Σ | 1 / 2 e − 1 2 ( � x − � x − � µ ) f � X ( � x ) = Linear algebra: A � v = λ� | A − λ I | = 0 , v