Learning Expressive Ontological Concept Descriptions via Neural - PowerPoint PPT Presentation

Framing the Problem: the Transformation Function “syntactic transformation of natural language definitions into description logic axioms.” (V¨ olker J., 2008) ı Ù produces honey bee insect ÷ . . an produces honey A bee is insect that All the extralogical symbols come from the sentence. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Framing the Problem syntactic transformation Descriptive ALCQ Language We need: datasets; architecture; MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Structure and Meaning Machine Learning means examples, good examples, many examples. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Structure and Meaning Machine Learning means examples, good examples, many examples. Every bee is an insect and it also produces honey. A bee is an insect that produces honey. Bees are insects that produce also honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Structure and Meaning Machine Learning means examples, good examples, many examples. Every bee is an insect and it also produces honey. A bee is an insect that produces honey. Bees are insects that produce also honey. Every bee is an insect that produces also honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Structure and Meaning Machine Learning means examples, good examples, many examples. Every bee is an insect and it also produces honey. A bee is an insect that produces Bees are insects that produce also honey. Every bee is an insect that produces also honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Structure and Meaning Machine Learning means examples, good examples, many examples. Every bee is an insect and it also produces honey. A bee is an insect that produces honey. Bees are insects that produce also honey. Every bee is an insect that produces also honey. Many structures, one meaning. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Meaning and Structure Machine Learning means examples, good examples, many examples. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Meaning and Structure Machine Learning means examples, good examples, many examples. Bees are insects that produce honey. A bee is also an insect that produces honey. Every bee is an insect and it also produces honey. A cow is a mammal that eats grass. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Meaning and Structure Machine Learning means examples, good examples, many examples. Bees are insects that produce honey. A bee is also an insect that produces honey. Every bee is an insect and it also produces honey. A cow is a mammal that eats grass. A cow is a mammal that produces milk. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Meaning and Structure Machine Learning means examples, good examples, many examples. Bees are insects that produce honey. A bee is also an insect that produces honey. Every bee is an insect and it also produces honey. A cow is a mammal that eats grass. A cow is a mammal that produces milk. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Meaning and Structure Machine Learning means examples, good examples, many examples. Bees are insects that produce honey. A bee is also an insect that produces honey. Every bee is an insect and it also produces honey. A cow is a mammal that eats grass. A cow is a mammal that produces milk. Many meanings, one structure. 2 2Other semantic phenomena are outside the scope of a syntactic transformation approach. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset Desiderata for the dataset: 3 covers many syntactic constructs (structure); covers many domains (meaning); has annotated < sentence, axiom > pairs. 3 G. Petrucci. “Information Extraction for Learning Expressive Ontologies” , ESWC 2015 Ph.D. Symp. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset Desiderata for the dataset: 3 covers many syntactic constructs (structure); covers many domains (meaning); has annotated < sentence, axiom > pairs. List of suitable datasets: 3 G. Petrucci. “Information Extraction for Learning Expressive Ontologies” , ESWC 2015 Ph.D. Symp. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset Desiderata for the dataset: 3 covers many syntactic constructs (structure); covers many domains (meaning); has annotated < sentence, axiom > pairs. List of suitable datasets: ? Following other notable approaches in literature, we started building a dataset to train our model. 3 G. Petrucci. “Information Extraction for Learning Expressive Ontologies” , ESWC 2015 Ph.D. Symp. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset ı Ù produces honey bee insect ÷ . . an produces honey A bee is insect that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset grass ı cow Ù ÷ eats mammal . . grass cow an eats A is mammal that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP A NP is a NP that VB NP C0 ı C1 Ù ÷ R0 . C2 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

First Challenge: the Dataset ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP A NP is a NP that VB NP C0 ı C1 Ù ÷ R0 . C2 Templates: structural regularities beyond meaning. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Context-Free Grammar V Every C0 R0 at least NUM C1 C0 ı > NUM R0 . C1 Every PhD student de- PhD student ı fends at least 1 thesis. > 1 defends . thesis MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Context-Free Grammar V Every C0 R0 at least NUM C1 C0 ı > NUM R0 . C1 Every PhD student de- PhD student ı fends at least 1 thesis. > 1 defends . thesis MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Context-Free Grammar V Every C0 R0 at least NUM C1 C0 ı > NUM R0 . C1 Every PhD student de- PhD student ı fends at least 1 thesis. > 1 defends . thesis MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Context-Free Grammar V random! Every C0 R0 at least NUM C1 C0 ı > NUM R0 . C1 Every innocent exile craves innocent exile ı at least 100 towers. > 100 craves . towers MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; A C0 is a C1 actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; A cow is a mammal sampling... A bee is an insect and parsing. A shark is a fish MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; A C0 is a C1 actualization; (repeat); approximation; C0 R0 C1 sampling... and parsing. C0 as also C1 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process ∼ Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process ∼ Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process ∼ Descriptive Language Context-Free Grammar template; actualization; (repeat); approximation; sampling... and parsing. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Data Generation Process Minimal input preprocessing: lower-cased; “an” → “a” ; “doesn’t” , “does not” , “don’t” → “do not” ; lemmatised nouns and verbs; numbers → NUM ; MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Second Challenge: the Model A 2-chapter adventure in the world of Recurrent Neural Networks: 2014-2015 Tag&Transduce G. Petrucci, C. Ghidini, and M. Rospocher “Ontology Learning in the Deep” EKAW 2016 2016-2017 Translate G. Petrucci, M. Rospocher, and C. Ghidini “Expressive Ontology Learning as Neural Machine Translation Task” (Under review) 4 4 Code & Datasets: https://github.com/dkmfbk/dket MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP Transduction from sentence to formula template; MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing ı Ù C0 C1 ÷ R0 C2 . . an A is that NP NP VB NP Transduction from sentence to formula template; Tagging extralogical symbols at the right place; MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing A bee is an in- sect that produces honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing C0 ı C1 Ù ÷ R0 . C2 ; transduction (F) A bee is an in- sect that produces honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing C0 ı C1 Ù ÷ R0 . C2 ; transduction (F) ? A bee is an in- Bee ı Insect sect that produces Ù÷ produces . Honey honey. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing C0 ı C1 Ù ÷ R0 . C2 ; transduction (F) ? A bee is an in- Bee ı Insect sect that produces Ù÷ produces . Honey honey. tagging (T) A bee C 0 is an insect C 1 that produces R 0 honey C 2 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing C0 ı C1 Ù ÷ R0 . C2 ; transduction (F) A bee is an in- Bee ı Insect sect that produces × Ù÷ produces . Honey honey. tagging (T) A bee C 0 is an insect C 1 that produces R 0 honey C 2 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

The Transducing Network Prediction ... C 0 ı <EOS> y k = argmax ( y k ) y 1 y 2 y m Output . . . y j = softmax ( Wh j + b ) h 1 h 2 h m Decoding . . . h j = g ( c , h j − 1 ) h n = c h 1 h 2 h 3 Hidden . . . h i = g ( x i , h i − 1 ) x 1 ± w x 2 ± w x 3 ± w x n ± w Embedding . . . x i = Ee i , Input . . . V ( A ) V ( bee ) V ( is ) V ( <EOS> ) V ( x i ) MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

The Tagging Network Tag . . . t 1 = w t 2 = C 0 t 3 = w t n = <EOS> y i = argmax ( y i ) y 1 y 2 y 3 y n Output . . . y i = softmax ( Wh i + b ) h 1 h 2 h 3 h n Hidden . . . h i = g ( x i , h i − 1 ) x 1 ± w x 2 ± w x 3 ± w x n ± w Embedding . . . x i = Ee i , Input-Windowing . . . V ( A ± w ) V ( bee ± w ) V ( is ± w ) V ( <EOS> ± w ) V ( x i ± w ) MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing: Evaluation RQ1. To what degree is the network capable to generalize over the syntactic structures of descriptive language? (many structures, one meaning) RQ2. To what degree is the network capable to tolerate words that have not been seen during the training phase? (many meanings, one structure) MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing: Evaluation Evaluation Metrics: Ó if f k ≡ ˆ f k 1 , q M 0 , otherwise k =1 FA ( ˆ F , F ) = CF Avg. Per-Formula Acc. = fully automated M M q M δ ( f k , ˆ f k ) ED ( ˆ k =1 Avg. Edit Distance F , F ) = semi-automated M Ó if f k = ˆ f k 1 , q Tf k q M j j j =1 k =1 0 , otherwise TA ( ˆ Avg. Per-Token Acc. F , F ) = quick control q M Tf k k =1 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing: Evaluation di ff erent training set sizes; 2M test examples; <UNK> between 20% and 40%. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing: Evaluation di ff erent training set sizes; 2M test examples; <UNK> between 20% and 40%. training set size CF FA ED TA 1000 10 0.5e-5 2.67 0.90 2000 8.05e-5 1.34 0.95 161 3000 60 3.00e-5 1.22 0.96 4000 22 1.10e-5 1.07 0.97 MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Tagging and Transducing: Evaluation di ff erent training set sizes; 2M test examples; <UNK> between 20% and 40%. training set size CF FA ED TA 1000 10 0.5e-5 2.67 0.90 2000 8.05e-5 1.34 0.95 161 3000 60 3.00e-5 1.22 0.96 4000 22 1.10e-5 1.07 0.97 Many limitations: we dropped the project and move forward. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Moving forward (aka 1 > 2) The placeholders are numbered in the training set and there is no way to overcome this limit—namely, generalize over the length of the sentence—by design. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Moving forward (aka 1 > 2) Negramaro is a red and strong wine. negramaro ı red wine Ù strong wine Negramaro is a red and strong wine. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Moving forward (aka 1 > 2) Negramaro is a red and strong wine. negramaro ı red wine Ù strong wine C0 ı C1 Û C2 transduction (F) Negramaro is a red ◊ and strong wine. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Moving forward (aka 1 > 2) Negramaro is a red and strong wine. negramaro ı red wine Ù strong wine C0 ı C1 Û C2 transduction (F) Negramaro is a red ◊ and strong wine. tagging (T) Negramaro C 0 is a red C 1 and strong C 2 wine C 2 . MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Moving forward (aka 1 > 2) Negramaro is a red and strong wine. negramaro ı red wine Ù strong wine C0 ı C1 Û C2 transduction (F) Negramaro is a red negramaro ı ◊ red Û strong wine and strong wine. tagging (T) Negramaro C 0 is a red C 1 and strong C 2 wine C 2 . MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate bee copy (#2) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı bee copy (#2) emit ( ı ) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı bee insect copy (#2) emit ( ı ) copy (#5) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı Ù bee insect copy (#2) emit ( ı ) copy (#5) emit ( Ù ) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı Ù bee insect ÷ copy (#2) emit ( ı ) copy (#5) emit ( Ù ) emit ( ÷ ) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı Ù produces bee insect ÷ copy (#2) emit ( ı ) copy (#5) emit ( Ù ) emit ( ÷ ) copy (#7) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı Ù produces bee insect ÷ . copy (#2) emit ( ı ) copy (#5) emit ( Ù ) emit ( ÷ ) copy (#7) emit ( . ) . an produces honey is insect A bee that MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate ı Ù produces honey bee insect ÷ . copy (#2) emit ( ı ) copy (#5) emit ( Ù ) emit ( ÷ ) copy (#7) emit ( . ) copy (#8) . an produces honey is insect A bee that Quasi-zero vocabulary setting. MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate Emit logical symbol Copy input word z t attention x 1 , x 2 , ..., x n MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate x T x x 1 x 2 x i ... ... MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate h i − 1 h T x − 1 h 2 h i h 1 ... ... g e ( x 1 , h 0 ) g e ( x 2 , h 1 ) g e ( x i , h i − 1 ) g e ( x T x , h T x ) x T x x 1 x 2 x i ... ... MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate d j − 1 Attention h T x h 1 h i h 2 h i − 1 h T x − 1 h 2 h i h 1 ... ... g e ( x 1 , h 0 ) g e ( x 2 , h 1 ) g e ( x i , h i − 1 ) g e ( x T x , h T x ) x T x x 1 x 2 x i ... ... MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER

Translate d j g d ( ..., d j − 1 ) ˜ y j − 1 d j − 1 c j d j − 1 Attention h T x h 1 h i h 2 h i − 1 h T x − 1 h 2 h i h 1 ... ... g e ( x 1 , h 0 ) g e ( x 2 , h 1 ) g e ( x i , h i − 1 ) g e ( x T x , h T x ) x T x x 1 x 2 x i ... ... MARCO Learning Expressive Ontological Concept Descriptions via Neural Networks ROSPOCHER