CS 730/830: Intro AI Unsuperv. Learning asst 11 posted Wheeler - PowerPoint PPT Presentation

CS 730/830: Intro AI Unsuperv. Learning asst 11 posted Wheeler Ruml (UNH) Lecture 23, CS 730 – 1 / 13

Unsuperv. Learning ■ Overview ■ Bottom-Up ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM ■ Basic Clustering Unsupervised Learning ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 2 / 13

Overview modeling = predicting = understanding Unsuperv. Learning clustering ■ Overview ■ Bottom-Up finding ‘structure’ in data ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM ■ Basic Clustering ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 3 / 13

Bottom-Up Unsupervised Learning explain the data all-at-once vs piece-by-piece? Unsuperv. Learning ■ Overview ■ Bottom-Up ■ RANSAC repeat ■ Break ■ k -Means make a model to explain a minimal amount of data ■ An Algorithm ■ EM check how much of the total data the model explains ■ Basic Clustering repeat until model fits a decent amount of the data ■ Summary ■ EOLQs when found, remove explained data from the set until hard to find a decent model or not enough data left Wheeler Ruml (UNH) Lecture 23, CS 730 – 4 / 13

Random Sample Consensus (RANSAC) given data, find a set of explanatory models: Unsuperv. Learning ■ Overview ■ Bottom-Up ■ RANSAC repeat ■ Break ■ k -Means repeat many times ■ An Algorithm randomly pick minimum data to fit model ■ EM ■ Basic Clustering find inliers ■ Summary repeat until no change ■ EOLQs fit model to inliers find new inliers if best model has enough inliers record model remove inliers from data until best model not good enough or not enough data left Wheeler Ruml (UNH) Lecture 23, CS 730 – 5 / 13

Break asst 10 ■ Unsuperv. Learning asst 11 ■ Overview ■ ■ Bottom-Up projects: four weeks from yesterday! ■ ■ RANSAC ■ Break Tue May 5: 9-noon: project presentations ■ ■ k -Means Mon May 11 2pm: final paper PDF and project tarball via ■ ■ An Algorithm ■ EM email ■ Basic Clustering ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 6 / 13

k -Means Clustering Naive Bayes model: choose class, generate attributes Unsuperv. Learning independently ■ Overview ■ Bottom-Up ■ RANSAC ■ Break mixture model: choose class, generate data ■ k -Means ■ An Algorithm ■ EM � P ( x | θ ) = P ( C = k | θ k ) P ( x | C = k, θ k ) ■ Basic Clustering ■ Summary k ■ EOLQs eg, for mixture of Gaussians, � � − ( x − µ k ) 2 1 P ( x | C = k, µ k , σ 2 k ) = exp 2 σ 2 � 2 σ 2 k π k Wheeler Ruml (UNH) Lecture 23, CS 730 – 7 / 13

An Algorithm Means represent the center of a cluster/class Unsuperv. Learning Values for the means are the model ■ Overview ■ Bottom-Up Model changes based on the classes assigned to the data ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM init the k means somehow ■ Basic Clustering repeat until cluster assignments do not change: ■ Summary ■ EOLQs Assign each data point to the mean nearest to it Calculate new means for the data assigned to each cluster Wheeler Ruml (UNH) Lecture 23, CS 730 – 8 / 13

An Algorithm Means represent the center of a cluster/class Unsuperv. Learning Values for the means are the model ■ Overview ■ Bottom-Up Model changes based on the classes assigned to the data ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM init the k means somehow ■ Basic Clustering repeat until cluster assignments do not change: ■ Summary ■ EOLQs Assign each data point to the mean nearest to it Calculate new means for the data assigned to each cluster Example Wheeler Ruml (UNH) Lecture 23, CS 730 – 8 / 13

An Algorithm Means represent the center of a cluster/class Unsuperv. Learning Values for the means are the model ■ Overview ■ Bottom-Up Model changes based on the classes assigned to the data ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM init the k means somehow ■ Basic Clustering repeat until cluster assignments do not change: ■ Summary ■ EOLQs Assign each data point to the mean nearest to it Calculate new means for the data assigned to each cluster Example Is the classification optimal? What is it optimizing? Wheeler Ruml (UNH) Lecture 23, CS 730 – 8 / 13

Expectation-Maximization model parameters θ (eg, µ, σ 2 , P ( C = k ) ) Unsuperv. Learning ■ Overview observed variables x j ■ Bottom-Up hidden variables C j ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM init the θ k somehow ■ Basic Clustering repeat until done: ■ Summary ■ EOLQs E: compute expected values of hidden vars: P ( C j = k | x j , θ k ) eg by αP ( C = k ) P ( x j | C = k, θ k ) M: maximize data likelihood using current estimates: θ k , with each x j weighted by P ( C j = k | x j ) , eg by � θ ← argmax P ( Z = z | x, θ ) P ( x, Z = z | θ ) θ z Wheeler Ruml (UNH) Lecture 23, CS 730 – 9 / 13

Expectation-Maximization model parameters θ (eg, µ, σ 2 , P ( C = k ) ) Unsuperv. Learning ■ Overview observed variables x j ■ Bottom-Up hidden variables C j ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM init the θ k somehow ■ Basic Clustering repeat until done: ■ Summary ■ EOLQs E: compute expected values of hidden vars: P ( C j = k | x j , θ k ) eg by αP ( C = k ) P ( x j | C = k, θ k ) M: maximize data likelihood using current estimates: θ k , with each x j weighted by P ( C j = k | x j ) , eg by � θ ← argmax P ( Z = z | x, θ ) P ( x, Z = z | θ ) θ z greedy increase of data likelihood Wheeler Ruml (UNH) Lecture 23, CS 730 – 9 / 13

Expectation-Maximization Features Unsuperv. Learning ■ Overview Probabilistic clustering ■ ■ Bottom-Up ■ RANSAC Explicit model ■ ■ Break Locally optimal ■ ■ k -Means ■ An Algorithm Issues ■ EM ■ Basic Clustering Number of classes (means, Gaussians, etc.) ■ Summary ■ ■ EOLQs Local maxima ■ Wheeler Ruml (UNH) Lecture 23, CS 730 – 10 / 13

Agglomerative Clustering dendrogram Unsuperv. Learning O ( n 2 ) vs O ( kn ) ■ Overview ■ Bottom-Up AutoClass ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM ■ Basic Clustering ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 11 / 13

Summary supervised learning: learning a function or a density Unsuperv. Learning unsupervised learning: explaining data ■ Overview ■ Bottom-Up reinforcement learning: learning how to act ■ RANSAC ■ Break ■ k -Means ■ An Algorithm ■ EM ■ Basic Clustering ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 12 / 13

EOLQs What question didn’t you get to ask today? ■ Unsuperv. Learning What’s still confusing? ■ Overview ■ ■ Bottom-Up What would you like to hear more about? ■ ■ RANSAC ■ Break Please write down your most pressing question about AI and put ■ k -Means ■ An Algorithm it in the box on your way out. ■ EM ■ Basic Clustering Thanks! ■ Summary ■ EOLQs Wheeler Ruml (UNH) Lecture 23, CS 730 – 13 / 13