learning effective and interpretable semantic models
play

Learning Effective and Interpretable Semantic Models using - PowerPoint PPT Presentation

Feb 16, 2023 •160 likes •1.03k views

1 Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE) Brian Murphy, Partha Talukdar, Tom Mitchell Machine Learning Department Carnegie Mellon University 2 Distributional Semantic Modeling 2

  1. 1 Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE) Brian Murphy, Partha Talukdar, Tom Mitchell Machine Learning Department Carnegie Mellon University

  2. 2 Distributional Semantic Modeling

  3. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space

  4. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ...

  5. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ... • Although effective, these models are often not interpretable

  6. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ... • Although effective, these models are often not interpretable Examples of top 5 words from 5 randomly chosen dimensions from SVD 300

  7. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008)

  8. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) “motorbike” Input stimulus word

  9. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) “motorbike” . . . (0.00, rub) Input (0.00, taste) stimulus word Semantic representation

  10. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) “motorbike” . . . (0.00, rub) Input (0.00, taste) stimulus word Semantic Mapping learned representation from fMRI data

  11. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) predicted “motorbike” . activity . for . “motorbike” (0.00, rub) Input (0.00, taste) stimulus word Semantic Mapping learned representation from fMRI data

  12. 4 Why interpretable dimensions? (Mitchell et al., Science 2008)

  13. 4 Why interpretable dimensions? (Mitchell et al., Science 2008)

  14. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns!

  15. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs

  16. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs ‣ can’t represent arbitrary concepts

  17. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs ‣ can’t represent arbitrary concepts ‣ need data-driven, broad coverage semantic representations

  18. 5 What is an Interpretable, Cognitively-plausible Representation?

  19. 5 What is an Interpretable, Cognitively-plausible Representation? features words

  20. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ...

  21. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ... 1. Compact representation: Sparse , many zeros

  22. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ... 1. Compact representation: 2. Uneconomical to store negative Sparse , many zeros (or inferable) characteristics: Non-Negative

  23. 5 What is an Interpretable, Cognitively-plausible Representation? features words 3. Meaningful Dimensions: w ... 0 0 0 ... 1.2 ... Coherent 1. Compact representation: 2. Uneconomical to store negative Sparse , many zeros (or inferable) characteristics: Non-Negative

  24. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative

  25. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned

  26. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing)

  27. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Hand Coded 83.5 Corpus-derived Corpus-derived 83.1 (existing) Prediction accuracy (on Neurosemantic Decoding) (Murphy, Talukdar, Mitchell, StarSem 2012)

  28. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing)

  29. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing) NNSE Our proposal

  30. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing) NNSE Our proposal

  31. 7 Non-Negative Sparse Embedding (NNSE)

  32. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X

  33. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X w i input representation for word w i

  34. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X A D = x w i input representation for word w i

  35. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i

  36. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative

  37. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative • sparsity penalty on the rows of A

  38. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative • sparsity penalty on the rows of A • alternating minimization between A and D, using SPAMS package

  39. 8 NNSE Optimization X A D = x

  40. 9 Experiments

  41. 9 Experiments • Three main questions

  42. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice?

  43. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE?

  44. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent?

  45. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup

  46. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup • partial ClueWeb09, 16bn tokens, 540m sentences, 50m documents

  47. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup • partial ClueWeb09, 16bn tokens, 540m sentences, 50m documents • dependency parsed using Malt parser

  48. 10 Baseline Representation: SVD

  49. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract

  50. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence

  51. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus

  52. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline

  53. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline • This is also the input (X) to NNSE

  54. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline • This is also the input (X) to NNSE • Other representations were also compared (e.g., LDA, Collobert and Weston, etc.), details in the paper

  55. 11 Is NNSE effective in Neurosemantic Decoding?

Recommend

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead Cynthia Rudin, Duke University Presenters : Sreya Dutta Roy, Ziqian Lin 1 Overview Introduction Explainable ML Vs

1.04k views • 39 slides
From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box

From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box

From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box Models Huge volumes of data Solve task Computing power Deep Neural Network Information (implicit) ICIP18 Tutorial on Interpretable Deep

540 views • 33 slides
Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker,

Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker,

Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker, Gerg o Bohner, Julien Boussard, Maneesh Sahani Gatsby Computational Neuroscience Unit University College London ICML June 12, 2019 nonlinear

671 views • 32 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila

Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila

Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila McIlraith 1 , 2 1 Department of Computer Science, University of Toronto 2 Vector Institute { acamacho, sheila } @cs.toronto.edu ICAPS 2019 July 13,

457 views • 19 slides
IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules

IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules

IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules Bishwamittra Ghosh Joint work with Kuldeep S. Meel 1 Applications of Machine Learning 2 Example Dataset 3 Representation of an interpretable

1.24k views • 55 slides
Toward Predictive Digital Twins via component-based reduced-order models and interpretable

Toward Predictive Digital Twins via component-based reduced-order models and interpretable

Toward Predictive Digital Twins via component-based reduced-order models and interpretable machine learning Michael Kapteyn*, Dr. David Knezevic, Prof. Karen Willcox AIAA SciTech | Paper AIAA-2020-0418 | January 6, 2020 Outline 1 Motivation

426 views • 27 slides
Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John

Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John

Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John Hopcroft Center Shanghai Jiao Tong University Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu, "Interpretable Convolutional Neural Networks"

694 views • 42 slides
Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

LRP Ariyan Zarei Motivation Having More interpretable Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks to have more interpretable Papers and Demo Introduction Machine Learning models

373 views • 35 slides
Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model el U-Tr Trees Guiliang Liu, Oliver Schulte, Wang Zhu, Qingcan Li Machine Learning Lab, ECML-PKDD 2018 Presentation PROBLEM DEFINITION

372 views • 15 slides
Ralph Hodgson Realizing a semantic solution: Ontologies are like and unlike other IT models

Ralph Hodgson Realizing a semantic solution: Ontologies are like and unlike other IT models

Enterprise Architecture in SOA: Coverage Models and Methodologies Going Semantic Semantic technology May 23 rd , 2005 Enterprise architecture and semantic technology Enterprise architecture maturity model Capabilities of semantic

244 views • 9 slides
RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic

RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic

RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic Web does not mean computers are going to understand the meaning of everything on the web does mean that we can create some standards ways of

344 views • 33 slides
Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth

Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth

Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth Gupta Sai Krishna Prasad Guided By:- Amitabha Mukherjee Distributional Semantic Models (DSMs) Distributional hypothesis - words that occur in

375 views • 11 slides
Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine

Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine

Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine Learning Models to Predict Health Outcomes Chris Fernandez 1,2 Sam Rusk 1,2 Nick Glattard ,1,2 Mehdi Shokoueinejad. 1,3 1 Department of Population Health

609 views • 21 slides
Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Fraunhofer Image Processing Heinrich Hertz Institute Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech Samek Northern Lights Deep Learning Workshop (NLDL19) Troms, Norway 8th January 2019

1.15k views • 51 slides
Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton

Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton

Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton Greenside, Anshul Kundaje Peyton Anshul 1 With great power comes really poor interpretability Deep Interpretable Deep Power Learning Learning

783 views • 41 slides
Self-supervised Learning of Interpretable Keypoints from Unlabelled Videos

Self-supervised Learning of Interpretable Keypoints from Unlabelled Videos

Self-supervised Learning of Interpretable Keypoints from Unlabelled Videos www.robots.ox.ac.uk/~vgg/research/ Oral presentation, CVPR 2020 unsupervised_pose/ Tomas Jakab Ankush Gupta VGG, University of Oxford DeepMind, London Hakan Bilen

259 views • 14 slides
Towards Interpretable Deep Learning for Natural Language Processing Roy Schwartz University of

Towards Interpretable Deep Learning for Natural Language Processing Roy Schwartz University of

Towards Interpretable Deep Learning for Natural Language Processing Roy Schwartz University of Washington & Allen Institute for Artificial Intelligence December 2018 1 / 37 ( Deep-Learning -Based) AI Today 2 / 37 3 / 37 Deep Learning

1.63k views • 140 slides
InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu,

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations Chih-Hui Ho, Chun Hu, Po-Jung Lai 1 Outline 1. Introduction 2. Related work Generative adversarial imitation learning (GAIL) 3. Proposed method 4. Experiment

692 views • 14 slides
Application: Semantic Role Labeling CS 6956: Deep Learning for NLP Overview What is semantic

Application: Semantic Role Labeling CS 6956: Deep Learning for NLP Overview What is semantic

Application: Semantic Role Labeling CS 6956: Deep Learning for NLP Overview What is semantic role labeling? The state-of-the-art before neural networks Neural models for semantic roles 1 Overview What is semantic role labeling?

930 views • 66 slides
Not Just a Black Box: Interpretable Deep Learning for Genomics Presented by: AvanA Shrikumar 1

Not Just a Black Box: Interpretable Deep Learning for Genomics Presented by: AvanA Shrikumar 1

Not Just a Black Box: Interpretable Deep Learning for Genomics Presented by: AvanA Shrikumar 1 With great power comes really poor interpretability Deep Power Learning Traditional machine learning Classical statistics 2

984 views • 82 slides
Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder Ryo

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder Ryo

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder Ryo Takahashi* 1 Ran Tian* 1 Kentaro Inui 1,2 (* equal contribution) 1 Tohoku University 2 RIKEN, Japan Task: Knowledge Base Completion Knowledge Bases

261 views • 23 slides
Effective Hamiltonians of anyon anyon lattice lattice Effective Hamiltonians of models models

Effective Hamiltonians of anyon anyon lattice lattice Effective Hamiltonians of models models

Effective Hamiltonians of anyon anyon lattice lattice Effective Hamiltonians of models models Alexander Protogenov Institute of Applied Physics of the RAS, Nizhny Novgorod, Russia with Luigi Martina Dipartimento di Fisica, Universit di

953 views • 26 slides
Incremental Approach to Interpretable Classification Rule Learning Bishwamittra Ghosh and Kuldeep

Incremental Approach to Interpretable Classification Rule Learning Bishwamittra Ghosh and Kuldeep

Incremental Approach to Interpretable Classification Rule Learning Bishwamittra Ghosh and Kuldeep S. Meel School of Computing, National University of Singapore CP 2019 Bishwamittra Ghosh Incremental Approach to Interpretable Classification Rule

267 views • 23 slides

More recommend