Ev aluating Hyp otheses �Read Ch� �� �Recommended exercises� ���� ���� ���� � Sample error� true error � Con�dence in terv als for observ ed h yp othesis error � Estimators � Binomial distribution� Normal distribution� Cen tral Limit Theorem � P aired tests t � Comparing learning metho ds �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Tw o De�nitions of Error The of h yp othesis with resp ect to true error h target function and distribution D is the f probabilit y that will misclassify an instance h dra wn at random according to D � er r or � h � � Pr � f � x � � � h � x �� D x �D The of with resp ect to target sample error h function and data sample is the prop ortion of f S examples misclassi�es h � � h � � � f � x � � � h � x �� er r or � X S n x � S Where � f � x � � � h � x �� is � if � x � � � h � x �� and � � f otherwise� Ho w w ell do es � h � estimate � h �� er r or er r or S D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Problems Estimating Error �� If is training set� � h � is Bias� S er r or S optimisticall y biased � � er � h �� � � h � bias E r or er r or S D F or un biased estimate� and m ust b e c hosen h S indep enden tly �� Ev en with un biased S � er r or � h � ma y V arianc e� S still from er r or � h � vary D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Example Hyp othesis misclassi�es �� of the �� examples in h S �� � h � � � � �� er r or S �� What is � h �� er r or D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Estimators Exp erimen t� �� c ho ose sample of size according to S n distribution D �� measure er r or � h � S � h � is a random v ariable �i�e�� result of an er r or S exp erimen t� � h � is an un biased for � h � er r or estimator er r or S D Giv en observ ed � h � what can w e conclude er r or S ab out � h �� er r or D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Con�dence In terv als If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately ��� probabilit y � � h � er r or D lies in in terv al v � h ��� � � h �� er r or er r or u u S S er r or � h � � � � �� u u S t n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Con�dence In terv als If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately N� probabilit y � � h � er r or D lies in in terv al v � h ��� � � h �� er r or er r or u u S S er r or � h � � z u u S N t n where N �� ��� ��� ��� ��� ��� ��� ��� � ���� ���� ���� ���� ���� ���� ���� z N �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
� h � is a Random V ariable er r or S Rerun the exp erimen t with di�eren t randomly dra wn S �of size n � Probabilit y of observing misclassi�ed examples� r Binomial distribution for n = 40, p = 0.3 0.14 0.12 0.1 0.08 P(r) 0.06 0.04 0.02 0 0 5 10 15 20 25 30 35 40 n � r n � r � r � � � h � �� � � h �� P er r or er r or D D r �� n � r �� �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Binomial Probabilit y Distributi on Binomial distribution for n = 40, p = 0.3 0.14 0.12 0.1 0.08 P(r) 0.06 0.04 0.02 0 0 5 10 15 20 25 30 35 40 n � r n � r � r � � �� � p � P p �� n � �� r r Probabilit y P � r � of r heads in n coin �ips� if p � Pr � heads � � Exp ected� or mean v alue of X � E � X �� is n � X � � � i � � E iP np X i �� � V ariance of is X � � X � � �� X � � X �� � � np �� � p � V ar E E � Standard deviation of X � � � is X r r � � � E �� X � E � X �� � � np �� � p � X �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Normal Distributi on Appro ximates Bino� mial � h � follo ws a distribution� with er r or Binomial S � mean � � h � � er r or er r or � h � D S � standard deviation � er r or � h � S v � h ��� � � h �� er r or er r or u u D D � u � u � h � er r or t S n Appro ximate this b y a distribution with Normal � mean � � h � � er r or er r or � h � D S � standard deviation � � h � er r or S v er r or � h ��� � er r or � h �� u u S S � � u u � h � er r or t S n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Normal Probabilit y Distributi on Normal distribution with mean 0, standard deviation 1 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 -3 -2 -1 0 1 2 3 � x � � � � � � � p p � x � � e � � � � � � The probabilit y that will fall in to the in terv al X � a� b � is giv en b y b Z p � x � dx a � Exp ected� or mean v alue of � � X �� is X E � X � � E � � V ariance of is X � � X � � V ar � � Standard deviation of � � is X � X � � � X �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Normal Probabilit y Distributi on 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 -3 -2 -1 0 1 2 3 ��� of area �probabilit y� lies in � � � �� � � N� of area �probabilit y� lies in � � z � N �� ��� ��� ��� ��� ��� ��� ��� N � ���� ���� ���� ���� ���� ���� ���� z N �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Con�dence In terv als� More Correctly If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately ��� probabilit y � � h � er r or S lies in in terv al v � h ��� � � h �� er r or er r or u u D D � h � � � � �� u er r or u D t n equiv alen tl y � er r or � h � lies in in terv al D v er r or � h ��� � er r or � h �� u u D D � h � � � � �� u er r or u S t n whic h is appro ximately v � h ��� � � h �� er r or er r or u u S S � h � � � � �� u er r or u S t n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Cen tral Limit Theorem Consider a set of indep enden t� iden ticall y distributed random v ariables � all go v erned Y � � � Y � n b y an arbitrary probabilit y distribution with mean � and �nite v ariance � De�ne the sample mean� � � � n � � Y Y X i n i �� As � � � the Cen tral Limit Theorem� n � distribution go v erning approac hes a Normal Y � � distribution� with mean and v ariance � � n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����
Recommend
More recommend