Review of Probability 1 Probability Theory: Many techniques in - PowerPoint PPT Presentation

Review of Probability 1

Probability Theory:  Many techniques in speech processing require the manipulation of probabilities and statistics.  The two principal application areas we will encounter are:  Statistical pattern recognition.  Modeling of linear systems. 2

Events:  It is customary to refer to the probability of an event.  An event is a certain set of possible outcomes of an experiment or trial.  Outcomes are assumed to be mutually exclusive and, taken together, to cover all possibilities. 3

Axioms of Probability:  To any event A we can assign a number, P(A), which satisfies the following axioms:  P(A) ≥ 0.  P(S) =1.  If A and B are mutually exclusive, then P(A+B) = P(A)+P(B).  The number P(A) is called the probability of A . 4

Axioms of Probability (some consequence) :  Some immediate consequence:  If is the complement of A, then A   ( ) A A S    ( ) 1 ( ) P A P A   P(0) ,the probability of the impossible event , is 0.  P(A) ≤ 1.  If two event A and B are not mutually exclusive, we can show that  P(A+B)=P(A)+P(B)-P(AB). 5

Conditional Probability:  The conditional probability of an event A , given that event B has occurred, is defined as: ( ) P AB  ( | ) P A B ( ) P B  We can infer P(B|A) by means of Bayes’ theorem: ( ) P B  ( | ) ( | ) P B A P A B ( ) P A 6

Independence:  Events A and B may have nothing to do with each other and they are said to be independent.  Two events are independent if P(AB)=P(A)P(B).  From the definition of conditional probability:  ( | ) ( ) P A B P A  ( | ) ( ) P B A P B     ( ) ( ) ( ) ( ) ( ) P A B P A P B P A P B 7

Independence:  Three events A,B and C are independent only if:   ( ) ( ) ( ) P AB P A P B    ( ) ( ) ( ) P AC P A P C   ( ) ( ) ( ) P BC P B P C     ( ) ( ) ( ) ( ) P ABC P A P B P C 8

Random Variables:  A random variable is a number chosen at random as the outcome of an experiment.  Random variable may be real or complex and may be discrete or continuous.  In S.P. ,the random variable encounter are most often real and discrete.  We can characterize a random variable by its probability distribution or by its probability density function (pdf). 9

Random Variables (distribution function) :  The distribution function for a random variable y is the probability that y does not exceed some value u ,   ( ) ( ) F y u P y u  and     ( ) ( ) ( ) P u y v F v F u y y 10

Random Variables (probability density function) :  The probability density function is the derivative of the distribution : d  ( ) ( ) f u F u y y du  v    ( ) ( )  and, P u y v f y dy y u   ( ) 1 F  y    ( ) 1 f y y dy    11

Random Variables (expected value) :  We can also characterize a random variable by its statistics.  The expected value of g(x) is written E{g(x)} or <g(x)> and defined as  Continuous random variable:     ( ) ( ) ( ) g x g x f x dx    Discrete random variable:    ( ) ( ) ( ) g x g x p x x 12

Random Variables (moments) :  The statistics of greatest interest are the moment of X.  The kth moment of X is the expected value k X of .  For a discrete random variable:    k k ( ) m X x p x k x 13

Random Variables (mean & variance) : m  The first moment, ,is the mean of x . 1   Continuous:   ( ) X xf x dx   å  Discrete: m = X =< X >= xp ( x ) x  The second central moment, also known as the variance of p(x), is given by     2 2 ( ) ( ) x x p x x   2 m X 2 14

Random Variables … :  To estimate the statistics of a random variable, we repeat the experiment which generates the variable a large number of times.  If the experiment is run N times, then each value x will occur Np(x) times, thus N 1   ˆ k m x k i N  1 i N 1    ˆ x x i N  1 i 15

Random Variables (Uniform density) :  A random variable has a uniform density on the interval (a, b) if :   0 , x a         ( ) ( ) /( ), F X x x a b a a x b    1 , x b     1 /( ), b a a x b   ( ) f X x   0 , otherwise 1   b  2 2  ( ) a 12 16

Random Variables (Gaussian density) :  The Gaussian, or normal density function is given by: 1 2 2        2 ( ) / x ( ; , ) n x e   2 17

Random Variables ( … Gaussian density) :  The distribution function of a normal variable is:   x      ( ; , ) ( ; , ) N x n u du   If we define error function as 1   x   2 / 2 u ( ) erf x e du   2    Thus, 1 x    ( ; , ) ( ) N x erf   18

Two Random Variables:  If two random variables x and y are to be considered together, they can be described in terms of their joint probability density f(x, y) or, for discrete variables, p(x, y).  Two random variable are independent if  ( , ) ( ) ( ) p x y p x p y  19

Two Random Variables ( … Continue) :  Given a function g(x, y), its expected value is defined as:       ( , ) ( , ) ( , ) g x y g x y f x y dxdy  Continuous:        ( , ) ( , ) ( , ) g x y g x y p x y  Discrete: , x y  And joint moment for two discrete random variable is:   i j ( , ) m x y p x y ij , x y 20

Two Random Variables ( … Continue) :  Moments are estimated in practice by averaging repeated measurements: N 1   ˆ i j m x y   ij N   1  A measure of the dependence of two random variables is their correlation and the correlation of two variables is their joint second moment:    ( , ) m xy xyp x y 11 , x y 21

Two Random Variables ( … Continue) :  The joint second central moment of x , y is their covariance :       ( )( ) x x y y m x y 11 xy  If x and y are independent then their covariance is zero.  The correlation coefficient of x and y is their covariance normalized to their standard deviations:   xy r   xy x y 22

Two Random Variables ( … Gaussian Random Variable ) :  Two random variables x and y are jointly Gaussian if their density function is :     2 2 1 1 2 x rxy y         ( , ) exp n x y        2 2 2      2 ( 1 ) 2 r   2 1   r x x y y x y  Where   xy r   xy x y 23

Two Random Variables ( … Sum of Random Variables ) :  The expected value of the sum of two random variables is :        x y x y  This is true whether x and y are independent or not  And also we have :     cx c x      x x i i i i 24

Two Random Variables ( … Sum of Random Variable ) :  The variance of the sum of the two independent random variable is :       2 2 2  x y x y  If two random variable are independent , the probability density of their sum is the convolution of the densities of the individual variables :     ( ) ( ) ( ) f z f u f z u du  x y x y  Continuous:       ( ) ( ) ( ) p z p u p z u   Discrete: x y x y   u 25

Central Limit Theorem  Central Limit Theorem (informal paraphrase ): If many independent random variables are summed, the probability density function (pdf) of the sum tends toward the Gaussian density, no matter what their individual densities are. 26

Multivariate Normal Density  The normal density function can be generalized to any number of random variables.  Let X be the random vector, [ , ,..., ] Col X X X 1 2 n   1       / 2 1 / 2 n ( ) ( 2 ) | | exp ( ) N x R Q x x     2  Where      1 T ( ) ( ) ( ) Q x x x x R x x  The matrix R is the covariance matrix of X     (R is Positive-Definite) T ( )( ) R x x x x 27

Random Functions :  A random function is one arising as the outcome of an experiment.  Random functions do not need to be functions of time, but in all cases of interest to us they will be.  A discrete stochastic process is characterized by many probability density functions of the form, ( , , ,..., , , , ,..., ) p x x x x t t t t 1 2 3 1 2 3 n n 28