... Other types of activation functions ( net = w i x i ) x n w n - PDF document

M otivation � Knowledge discovery process FEATURE SELECTION USING Interpretation ANT COLONY OPTIM IZATION: M odeling Knowledge APPLICATIONS IN HEALTH CARE Feature selection Patterns Preprocessing Reduced data João M . C. Sousa 1 Data acquisition jmsousa@ist.utl.pt Preprocessed S. M . Vieira 1 , S. N. Finkelstein 2,3 , A. S. Fialho 1,2 , data T arget data Data . Cismondi 1,2 , S. R. Reti 3 and M . D. Howell 3 F 1 Technical University of Lisbon, Instituto Superior Técnico, Dept. of Mechanical Engineering, CIS/IDMEC – LAETA, Av. Rovisco Pais, 1049-001 Lisbon, Portugal From G. Piatetsky-Shapiro U. Fayyad and P . Smyth. From data mining to knowledge discovery in databases. 2 Massachusetts Institute of Technology, Engineering Systems Division, 77 Massachusetts Artificial Intelligence Magazine , 17(3):37-54, 1996. Avenue, 02139 Cambridge, MA, USA 3 Division of Clinical Informatics, Department of Medicine, Beth Israel Deaconess Medical Centre, Harvard Medical School, Boston, MA, USA 20 September 2010 Eindhoven, the Netherlands 2 Outline � M otivation � M odeling � Neural networks � Fuzzy sets and systems NEURAL NETWORKS � Fuzzy modeling � Feature selection � Ant colony optimization � Ant feature selection � Application: predicting outcomes of sepsis patients 20 September 2010 Eindhoven, the Netherlands 3 Artificial neuron Types of neurons � M cCulloch and Pits (1943) x 1 � Threshold � : � � n � � � � y sign � w x � x 2 w 2 y i i � � � i 1 Neuron ... � Other types of activation functions ( net = � w i x i ) x n w n 1 1 � x i : i- th input of the neuron 0 � w i : synaptic strength ( weight ) for x i � � � y = � ( � w i x i ): output signal 1, if net 0 linear � 1 � � step � y sigmoid y y net � � � � 0, if net 0 net 1 e Eindhoven, the Netherlands 5 Eindhoven, the Netherlands 6 1

M ulti-Layer Perceptron (M LP) M ost common M LP Hidden layer � Can learn functions that are not linearly separable. h b 1 Output layer h w 11 o b 1 o 1 w 11 x 1 y 1 1 ... Output signals ... 2 ... x i h w ij y k k o w jk ... j ... ... y l l x n h w nm o w ml o m b l h b m Eindhoven, the Netherlands 7 Eindhoven, the Netherlands 8 M ost common M LP Learning in NN � Biological neural networks: � Output of neurons in the hidden-layer h j : � � � � � Synaptic connections amongst neurons which � � � � n � � � n h h h h w x b w x simultaneously exhibit high activity are strengthned. � � j ij i j ij i i 1 i 0 � � � Artificial neural networks: � � n h tanh w x � � sigmoid � M athematical approximation of biological learning. � ij i i 0 � Error minimization ( nonlinear optimization problem). � Output of neurons in the output-layer y k : � Error backpropagation (first-order gradient) � � � � � Newton methods (second-order gradient) � � � � m � � � m o o o y w h b w h � � � Levenberg-M arquardt (second-order gradient) k jk j j jk k j 1 j 0 � Conjugate gradients � � m � � linear o w h � ... � jk j j 0 Eindhoven, the Netherlands 9 Eindhoven, the Netherlands 10 Supervised learning Bibliography e � S . Haykin. Neural Networks - A Comprehensive Foundation . Prentice Hall, 1999. � J.-S. Jang, C.-T. Sun and E. M izutani. Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence . Prentice Hall, New Jersey, 1997. � Andries P. Engelbrecht. Computational Intelligence: An x y Introduction . John Wiley, Chichester, 2002 � M ichael Negnevitsky. Artificial Intelligence: A Guide to T � � � � T T � T � Training data: X x x x � Intelligent Systems . Addison-Wesley, Pearson 1 2 N � � T � � Education, 2002. T T � T Y y y y � 1 2 N Eindhoven, the Netherlands 11 Eindhoven, the Netherlands 12 2

Introduction � How to simplify very complex systems? Allow some degree of uncertainty in their � description! FUZZY SETS � How to deal mathematically with uncertainty? Using probabilistic theory ( stochastic ). � Using the theory of fuzzy sets ( non-stochastic ). � Basic Concepts � Proposed in 1965 by Lotfi Zadeh (Fuzzy Sets, Information Control , 8, pp. 338-353). � Imprecision or vagueness in natural language does not imply a loss of accuracy or meaningfulness! Eindhoven, the Netherlands 14 Classical set Logic propositions � Example : set of old people A = { age | age � 70} � “Nick is old” ... true or false � Nick’s age: A 1 � age Nick = 70, � A (70) = 1 (true) � age Nick = 69.9, � A (69.9) = 0 (false) A 1 0.5 0.5 0 50 60 70 80 90 100 0 50 60 70 80 90 100 Eindhoven, the Netherlands 15 16 Fuzzy set Fuzzy proposition � “Nick is old”... degree of truth � G raded membership , element belongs to a set to a � A (70) = 0.5 certain degree. � age Nick = 70, � age Nick = 69.9, � A (69.9) = 0.49 A 1 � A (90) = 1 � age Nick = 90, membership grade A 1 membership grade 0.5 0.5 0 50 60 70 80 90 100 0 50 60 70 80 90 100 Eindhoven, the Netherlands 17 18 3

Typical linguistic values Linguistic variable � x is age � �� = { young , middle age , old } 1 young middle age old membership grade young middle age old 1 semantic rules M X membership grade 0 20 40 60 80 100 0 20 40 60 80 100 Eindhoven, the Netherlands 19 20 Fuzzy complement Intersection of fuzzy sets � � � ( x ) = 1 – � A ( x ) � � A � B ( x ) = min( � A ( x ), � B ( x )) � � 1 A A B x x Eindhoven, the Netherlands 21 Eindhoven, the Netherlands 22 Union of fuzzy sets � � A � B ( x ) = max( � A ( x ), � B ( x )) � FUZZY SYSTEM S A B x Eindhoven, the Netherlands 23 4

Linguistic variable Fuzzy if-then rules { x , �� , � , M X } � Fuzzy propositions � Where: � x is A , y is B � x – name of the linguistic variable � Linguistic (M amdani) fuzzy if-then rule: � ��� – linguistic values (terms) If x is A then y is B � �� – Universe of discourse � Antecedent : x is A � M X – semantic rule that associates each linguistic � Consequent : y is B value to a membership function. � Rule “ If x is A then y is B” is represented by a fuzzy relation defined on X � Y . Eindhoven, the Netherlands 25 Eindhoven, the Netherlands 26 Examples Linguistic (M amdani) model � If the road is slippery then brake softly. � If error is Negative big and � e is Positive big then � u is � � k k k R : If x is A then y is B , k 1,2, , K Negative small . � If a tomato is red then the tomato is ripe. � Decomposing using conjunctive forms: � If the temperature is very high then reduce the heat a � k k k k R : If x is A and x is A and and x is A 1 1 2 2 n n � lot. k k k then y is B and y is B and and y is B 1 1 2 2 p p � If the valve is closed then the pressure is high. � Degree of fulfillment of antecedents: � � � � � � � � k � � = ( ) x ( x ) ( x ), k 1,2, , K k 1 k 2 k n A A A 1 2 n Eindhoven, the Netherlands 27 Eindhoven, the Netherlands 28 Takagi-Sugeno fuzzy model Bibliography � G. Klir and T. Folger. Fuzzy S ets Uncertainty and Information . Prentice Hall, 1988. � � � k k k k R : If x is A then y f ( ), x k 1,2, , K � J.-S. Jang, C.-T. Sun and E. Mizutani. Neuro-Fuzzy and S oft Computing: A Computational Approach to Learning and � Affine linear form: Machine Intelligence . Prentice Hall, New Jersey, 1997. � � T � � k k k k k R : If x is A then y a x b � Andries P. Engelbrecht. Computational Intelligence: An Introduction . John Wiley, Chichester, 2002. � Degree of fulfillment � k defined as in linguistic models � J.M.C. Sousa and U. Kaymak. Fuzzy Decision Making in Modeling and Control . World Scientific Series in Robotics and Intelligent � M odel output given by the weighted fuzzy-mean: Systems, vol. 27. World Scientific Pub. Co., Singapore, Dec. 2002 � � � Michael Negnevitsky. Artificial Intelligence: A Guide to � � � K � K � � k k T k k k y ( a ) x b Intelligent S ystems . Addison-Wesley, Pearson Education, 2002. � � � k 1 y k 1 � � � R. Babuska. Fuzzy Modeling for Control . Kluwer Academic K � K � � j � j Publishers, 1998. j 1 j 1 Eindhoven, the Netherlands 29 Eindhoven, the Netherlands 30 5