Logistic Regression CMSC 678 UMBC Recap from last time Central - PowerPoint PPT Presentation

Maximum Entropy Models/ Logistic Regression CMSC 678 UMBC

Recap from last time…

Central Question: How Well Are We Doing? • Precision, This does Recall, F1 • Accuracy not have to • Log-loss be the same Classification • ROC-AUC thing as the • … loss • (Root) Mean Square Error function • Mean Absolute Error Regression • you … optimize Clustering • Mutual Information • V-score • the task : what kind … of problem are you solving?

Rule #1

We’ve only developed binary classifiers so far… Which option you choose is problem-dependent: Option 1: Develop a multi- class version 1. Why might you want to use option 1 or options Option 2: Build a one-vs-all OvA/AvA? (OvA) classifier 2. What are the benefits of Option 3: Build an all-vs-all OvA vs. AvA? (AvA) classifier 3. What if you start with a (there can be others) balanced dataset, e.g., 100 instances per class?

Some Classification Metrics Different ways of Accuracy averaging in a Trade-off and multi-class & multi- weight Precision label setting Recall AUC (Area Under Curve) Correct Value F1 # # # Confusion Matrix Guesse # # # d Value # # #

Outline Log-Linear (Maximum Entropy) Models Basic Modeling Connections to other techniques (“… by any other name…”) Objective to optimize Regularization

Maximum Entropy (Log-linear) Models 𝑞 𝑧 𝑦) ∝ exp(𝜄 𝑈 𝑔 𝑦, 𝑧 ) “model the posterior probabilities of the K classes via linear functions in θ , while at the same time ensuring that they sum to one and remain in [0, 1]” ~ Ch 4.4

Document Classification A TTACK Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region. Observed document Label Q: What features of this document could indicate an A TTACK ?

Document Classification Three people have been A TTACK fatally shot, and five A TTACK people, including a mayor, • # killed: were seriously wounded as a result of a Shining • Type : Path attack today against a attack community in Junin • Perp : department, central Peruvian mountain region.

Document Classification Three people have been fatally shot, and five A TTACK people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region. there could be many relevant clues

Features f fatally shot, ATTACK ( 🗏 , A TTACK ) The “clues” that help our system make its decision f seriously wounded, ATTACK ( 🗏 , A TTACK ) f Shining Path, ATTACK ( 🗏 , A TTACK ) Apply a vector of features f happy cat, ATTACK ( 🗏 , A TTACK ) 𝑔 🗏 , 𝑧 = (𝑔 1 ( 🗏 , 𝑧), … , 𝑔 𝐿 ( 🗏 , 𝑧)) … to a given document 🗏 and possible label y

Features The “clues” that help our system make its decision Apply a vector of features f fatally shot, ATTACK ( 🗏 , A TTACK ) 𝑔 🗏 , 𝑧 = (𝑔 1 ( 🗏 , 𝑧), … , 𝑔 𝐿 ( 🗏 , 𝑧)) f seriously wounded, ATTACK ( 🗏 , A TTACK ) to a given document 🗏 and f Shining Path, ATTACK ( 🗏 , A TTACK ) possible label y f happy cat, ATTACK ( 🗏 , A TTACK ) … Each feature function 𝑔 𝑙 can take any real value: binary count-based likelihood

Features The “clues” that help our system make its decision Apply a vector of features 𝑔 🗏 , 𝑧 = (𝑔 1 ( 🗏 , 𝑧), … , 𝑔 𝐿 ( 🗏 , 𝑧)) to f fatally shot, ATTACK ( 🗏 , A TTACK ) a given document 🗏 and possible f seriously wounded, ATTACK ( 🗏 , A TTACK ) label y f Shining Path, ATTACK ( 🗏 , A TTACK ) Each feature function 𝑔 𝑙 can take any real value: f happy cat, ATTACK ( 🗏 , A TTACK ) binary … count-based likelihood Features that don’t “ fire ” don’t apply to the pair 𝑙 🗏 , 𝑧 = 0 𝑔

Features: Score and Combine Our Possibilities θ fatally shot, ATTACK ( 🗏 , A TTACK ) θ seriously wounded, ATTACK ( 🗏 , A TTACK ) define for each key θ Shining Path, ATTACK ( 🗏 , A TTACK ) phrase/ clue... θ happy cat, ATTACK ( 🗏 , A TTACK ) … Remember: each θ w, l ( 🗏 ,y) is actually computed as θ w, l * f w, l ( 🗏 ,y)

Features: Score and Combine Our Possibilities … and for each label θ fatally shot, ATTACK ( 🗏 , A TTACK ) θ fatally shot, TECH ( 🗏 , A TTACK ) θ seriously wounded, ATTACK ( 🗏 , A TTACK ) θ seriously wounded, TECH ( 🗏 , A TTACK ) define for each key θ Shining Path, ATTACK ( 🗏 , A TTACK ) θ Shining Path, TECH ( 🗏 , A TTACK ) phrase/ clue... θ happy cat, ATTACK ( 🗏 , A TTACK ) θ happy cat, TECH ( 🗏 , A TTACK ) … … Remember: each θ w, l ( 🗏 ,y) is actually computed as θ w, l * f w, l ( 🗏 ,y)

Features: Score and Combine Our Possibilities … and for each label θ fatally shot, ATTACK ( 🗏 , A TTACK ) θ fatally shot, TECH ( 🗏 , A TTACK ) θ seriously wounded, ATTACK ( 🗏 , A TTACK ) θ seriously wounded, TECH ( 🗏 , A TTACK ) define for each key θ Shining Path, ATTACK ( 🗏 , A TTACK ) θ Shining Path, TECH ( 🗏 , A TTACK ) phrase/ clue... θ happy cat, ATTACK ( 🗏 , A TTACK ) θ happy cat, TECH ( 🗏 , A TTACK ) … … Remember: each Not all of these will be relevant θ w, l ( 🗏 ,y) is actually computed as θ w, l * f w, l ( 🗏 ,y)

Features: Score and Combine Our Possibilities … and for each label θ fatally shot, ATTACK ( 🗏 , A TTACK ) θ fatally shot, TECH ( 🗏 , A TTACK ) θ seriously wounded, ATTACK ( 🗏 , A TTACK ) θ seriously wounded, TECH ( 🗏 , A TTACK ) define for each key θ Shining Path, ATTACK ( 🗏 , A TTACK ) θ Shining Path, TECH ( 🗏 , A TTACK ) phrase/ clue... θ happy cat, ATTACK ( 🗏 , A TTACK ) θ happy cat, TECH ( 🗏 , A TTACK ) … … Remember: each Each of these scored features describes how “good” a θ w, l ( 🗏 ,y) is actually particular phrase is for a given document type if the computed as provided document document 🗏 has a proposed type θ w, l * f w, l ( 🗏 ,y)

Score and Combine Our Possibilities Shortcut notation: focus only on the features that “fire” Q : How many features are there? θ 1 (fatally shot, A TTACK ) θ 2 (seriously wounded, A TTACK ) θ 3 (Shining Path, A TTACK ) A : As many as you want there to be (but be … careful of underfitting/overfitting) Weight each of these: score how “important” each feature (clue) is

Score and Combine Our Possibilities θ 1 (fatally shot, A TTACK ) C OMBINE posterior θ 2 (seriously wounded, A TTACK ) probability of θ 3 (Shining Path, A TTACK ) A TTACK … Weight each of these: score how “important” each feature (clue) is

Scoring Our Possibilities Three people have been fatally shot, and five people, including a score( , ) = mayor, were seriously wounded as a result of a Shining Path attack A TTACK today against a community in Junin department, central Peruvian mountain region . θ 1 (fatally shot, A TTACK ) θ 2 (seriously wounded, A TTACK ) θ 3 (Shining Path, A TTACK ) … our linear regression model

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . Three people have been fatally shot, and five people, including a S NAP ( score( , ) ) mayor, were seriously wounded as a result of a Shining Path A TTACK attack today against a community in Junin department, central Peruvian mountain region .

What function… operates on any real number? is never less than 0?

What function… operates on any real number? is never less than 0? f(x) = exp(x)

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . Three people have been fatally shot, and five people, including a exp ( score( , ) ) mayor, were seriously wounded as a result of a Shining Path A TTACK attack today against a community in Junin department, central Peruvian mountain region .

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . θ 1 (fatally shot, A TTACK ) exp ( ) ) θ 2 (seriously wounded, A TTACK ) θ 3 (Shining Path, A TTACK ) … this is assuming binary features, but they don’t have to be

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . weight 1 * f 1 (fatally shot, A TTACK ) exp ( ) ) weight 2 * f 2 (seriously wounded, A TTACK ) weight 3 * f 3 (Shining Path, A TTACK ) …