Text Classification Contd + Document Representations Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 17, 2017 Based on slides from Nathan Schneider, Noah Smith, Dan Klein and everyone else they copied from.
Outline Logistic Regression Brief Intro to Neural Networks Document Representations CS 295: STATISTICAL NLP (WINTER 2017) 2
Outline Logistic Regression Brief Intro to Neural Networks Document Representations CS 295: STATISTICAL NLP (WINTER 2017) 3
Text Classification Paper Title CS Area • Human Computer Interaction Human machine interface for Theory • ABC computer applications • Artificial Intelligence Systems • CS 295: STATISTICAL NLP (WINTER 2017) 4
Linear Models Human machine interface for ABC computer applications CS 295: STATISTICAL NLP (WINTER 2017) 5
Matrix/Neural View CS 295: STATISTICAL NLP (WINTER 2017) 6
Naïve Bayes as a Linear Model CS 295: STATISTICAL NLP (WINTER 2017) 7
Joint vs Conditional Likelihood CS 295: STATISTICAL NLP (WINTER 2017) 8
Logistic Regression Model CS 295: STATISTICAL NLP (WINTER 2017) 9
Logistic Regression: 2 classes CS 295: STATISTICAL NLP (WINTER 2017) 10
Estimating the parameters CS 295: STATISTICAL NLP (WINTER 2017) 11
Gradient Descent CS 295: STATISTICAL NLP (WINTER 2017) 12
Tips and Tricks: TF-IDF Sparsity of Words Remember Zipf’s Law? Lots of rare words • For classification, they can be more informative! • CS 295: STATISTICAL NLP (WINTER 2017) 13
Tips and Tricks: TF-IDF Why use log(proportion) It works… • Importance is not a linear function • IDF is an additive function • CS 295: STATISTICAL NLP (WINTER 2017) 14
Tips and Tricks: Regularization Overfitting Training data is finite: thus has spurious correlations • Rare words that occur with one label! • Or don’t occur often enough • Curse of the Zipf’s Law continues… • For a word that occurs 10 times… There are many that occur ~10 times! CS 295: STATISTICAL NLP (WINTER 2017) 15
Tips and Tricks: Regularization Fixing Overfitting Ignore rare words (opposite of TF-IDF) • Penalize really high weights… • Accuracy Regularization Strength CS 295: STATISTICAL NLP (WINTER 2017) 16
Tips and Tricks: Featurizing CS 295: STATISTICAL NLP (WINTER 2017) 17
Outline Logistic Regression Brief Intro to Neural Networks Document Representations CS 295: STATISTICAL NLP (WINTER 2017) 18
Neural View of Log. Regression CS 295: STATISTICAL NLP (WINTER 2017) 19
Linear vs Non-linear Model CS 295: STATISTICAL NLP (WINTER 2017) 20
Introducing a Hidden Layer CS 295: STATISTICAL NLP (WINTER 2017) 21
What is Deep Learning? Many hidden layers In NLP, utilize unlabeled data to learn representations… (next lecture) CS 295: STATISTICAL NLP (WINTER 2017) 22
Outline Logistic Regression Brief Intro to Neural Networks Document Representations CS 295: STATISTICAL NLP (WINTER 2017) 23
Document Similarity A survey of user opinion of computer system response time Relation of user perceived response time to error measurement The generation of random, binary, ordered trees CS 295: STATISTICAL NLP (WINTER 2017) 24
Cosine Distance Advantages • Between -1 and 1 (0 means no overlap) If all >0, it is between 0 and 1 • • Size of vectors don’t matter CS 295: STATISTICAL NLP (WINTER 2017) 25
Term Document Matrix CS 295: STATISTICAL NLP (WINTER 2017) 26
Local and Global Weighting Local Weighting Global Weighting • Binary: • Binary: • Term Freq: • Normal: Log: IDF: • • CS 295: STATISTICAL NLP (WINTER 2017) 27
Example: Documents c1: Human machine interface for ABC computer applications c2: A survey of user opinion of computer system response time c3: The EPS user interface management system c4: System and human system engineering testing of EPS c5: Relation of user perceived response time to error measurement m1: The generation of random, binary, ordered trees m2: The intersection graph of paths in trees m3: Graph minors IV: Widths of trees and well-quasi-ordering m4: Graph minors: A survey From http://lsa.colorado.edu/papers/dp1.LSAintro.pdf CS 295: STATISTICAL NLP (WINTER 2017) 28
Example: Term-Doc Matrix c1 c2 c3 c4 c5 m1 m2 m3 m4 human interface computer user system response time EPS survey trees graph minors CS 295: STATISTICAL NLP (WINTER 2017) 29
Example: Distance Matrix c1 c2 c3 c4 c5 m1 m2 m3 m4 c1 c2 c3 c4 c5 m1 m2 m3 m4 CS 295: STATISTICAL NLP (WINTER 2017) 30
Problems with Sparse Vectors c2: A survey of user opinion of computer system response time c1: Human machine interface m4: Graph minors: A survey for ABC computer applications CS 295: STATISTICAL NLP (WINTER 2017) 31
Example: Distance Matrix c1 c2 c3 c4 c5 m1 m2 m3 m4 c1 c2 c3 c4 c5 m1 m2 m3 m4 CS 295: STATISTICAL NLP (WINTER 2017) 32
Option 1: Clustering CS 295: STATISTICAL NLP (WINTER 2017) 33
Example: Clustering c1 c1 c2 c2 c3 c3 c4 c4 c5 c5 m1 m1 m2 m2 m3 m3 m4 m4 CS 295: STATISTICAL NLP (WINTER 2017) 34
Upcoming… • Homework 1 is up! No more material will be covered Homework • • Due: January 26, 2017 Project pitch is due January 23, 2017! • Start assembling teams now Project • Tons of datasets on the “projects” page on website • CS 295: STATISTICAL NLP (WINTER 2017) 35
Recommend
More recommend
