combining data intense and compute intense methods for
play

Combining Data-Intense and Compute-Intense Methods for Fine-Grained - PowerPoint PPT Presentation

Combining Data-Intense and Compute-Intense Methods for Fine-Grained Morphological Analyses Petra Steiner Friedrich Schiller University Jena Jena, Germany September 19, 2019 Outline 3 September 19, 2019 Fine-Grained Morphological Analyses


  1. Combining Data-Intense Methods with Contextual Retrieval Chef rede:<>n:<> A:akte U:urin September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Chefredakteurin a. [[NN,NN],[NNSUFF]] (2) Chef redakt eur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Word Splitting and Contextual Retrieval Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> A:akte U:urin Chef R:redakteur in (1) and difgerent datasets with other morphological information Main lexicon with 42,205 entries, proper name lexicons with 16,718 entries Stuttgarter Morphological Analysis Tool, adjusted by the add-on Moremorph SMOR: A Morphological Tool for German 11 / 29 Example output for Chefredakteurin ‘editor-in-chief f emale ’: Chefredakteur | in b. ♯ [[NN],[NN, NNSUFF]] Chef | redakteurin c. ♯ [[NN, NN, NNSUFF]]

  2. Combining Data-Intense Methods with Contextual Retrieval Chef rede:<>n:<> A:akte U:urin September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Chefredakteurin a. [[NN,NN],[NNSUFF]] (2) Chef redakt eur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Word Splitting and Contextual Retrieval Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> A:akte U:urin Chef R:redakteur in (1) and difgerent datasets with other morphological information Main lexicon with 42,205 entries, proper name lexicons with 16,718 entries Stuttgarter Morphological Analysis Tool, adjusted by the add-on Moremorph SMOR: A Morphological Tool for German 11 / 29 Example output for Chefredakteurin ‘editor-in-chief f emale ’: Chefredakteur | in b. ♯ [[NN],[NN, NNSUFF]] Chef | redakteurin c. ♯ [[NN, NN, NNSUFF]]

  3. Combining Data-Intense Methods with Contextual Retrieval Chef rede:<>n:<> A:akte U:urin September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Chefredakteurin a. [[NN,NN],[NNSUFF]] (2) Chef redakt eur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Word Splitting and Contextual Retrieval Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> A:akte U:urin Chef R:redakteur in (1) and difgerent datasets with other morphological information Main lexicon with 42,205 entries, proper name lexicons with 16,718 entries Stuttgarter Morphological Analysis Tool, adjusted by the add-on Moremorph SMOR: A Morphological Tool for German 11 / 29 Example output for Chefredakteurin ‘editor-in-chief f emale ’: Chefredakteur | in b. ♯ [[NN],[NN, NNSUFF]] Chef | redakteurin c. ♯ [[NN, NN, NNSUFF]]

  4. Combining Data-Intense Methods with Contextual Retrieval Chef rede:<>n:<> A:akte U:urin September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Chefredakteurin a. [[NN,NN],[NNSUFF]] (2) Chef redakt eur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Word Splitting and Contextual Retrieval Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> akt eur in Chef rede:<>n:<> A:akt e U:urin Chef rede:<>n:<> A:akteur in Chef rede:<>n:<> A:akte U:urin Chef R:redakteur in (1) and difgerent datasets with other morphological information Main lexicon with 42,205 entries, proper name lexicons with 16,718 entries Stuttgarter Morphological Analysis Tool, adjusted by the add-on Moremorph SMOR: A Morphological Tool for German 11 / 29 Example output for Chefredakteurin ‘editor-in-chief f emale ’: Chefredakteur | in b. ♯ [[NN],[NN, NNSUFF]] Chef | redakteurin c. ♯ [[NN, NN, NNSUFF]]

  5. Combining Data-Intense Methods with Contextual Retrieval Recheck simple Word Splitting and Contextual Retrieval NN V NNSUFF NNSUFF no Build all combinations Filter out implausibles Contextual search in Wikipedia corpus analyses Moremorph Frequencies in Wikipedia corpus Weighting by word lengths Analyzable? yes no Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Chefredakteurin Chef redakt eur in SMOR & Lexical DBs Wordlists Bambussieb preußisch-europäisch Hybrid Word Analyzer Morphological Trees (*Aufwand* (*aufwenden* New splits Lexemes Monomorphemic Found in bases? Results : yes subanalyses Check for data- 12 / 29 Arbeitsaufwand (*Arbeit* arbeiten) | s | (*Aufwand* (*aufwenden* auf | wenden)) Arbeitsaufwand Chefredakteurin Chefredakteurin (*Chefredakteur* Chef | (*Redakteur* redakt | eur)) | in Bambus | Sieb (*Arbeit* arbeiten) | s | auf | wenden)) preußisch | - | europäisch Chefredakteur | in Chefredakteur | in ♯ Bambussieb ( ♯ preußisch | - | Europa | isch) Bambus | Sieb

  6. Combining Data-Intense Methods with Contextual Retrieval Text indices: for the tokenized and lemmatized forms. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 compensated by the frequencies of the other constituents of the split sequence. leads to a document frequency of 0 for this constituent, which can be (1) Contextual Search in Wikipedia Corpus 13 / 29 TreeTagger (Schmid, 1999) Corpus: 1.8 million articles of the annotated German Wikipedia Korpus of 2015 Tokenizer: a modifjed version of the tool from Dipper (2016); lemmatizer: Idea: For splits of unknown compounds, each immediate constituent should be found within the context at least somewhere inside a large corpus. For derivatives, this holds only for hypothetical constituents which are free morphs or lexemes. Contexts: the texts of a corpus in which the respective analyzed word form occurs. (Margaretha and Lüngen, 2014) For each text containing the input word form W wf , the document frequencies ( df 1 ...df m ) of the free hypothetical immediate constituents ( c wf ,s, 1 ...c wf ,s,n ) are being retrieved and summarized. This yields a text frequency score ( S wf ,s,t ) for each text and split of n constituents. n � S wf ,s,t = df i c =1 Of all morphological analyses for W wf , the one with the largest score is processed for the storage. A missing hypothetical constituent inside a text containing W wf

  7. Combining Data-Intense Methods with Contextual Retrieval Text indices: for the tokenized and lemmatized forms. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 compensated by the frequencies of the other constituents of the split sequence. leads to a document frequency of 0 for this constituent, which can be (1) Contextual Search in Wikipedia Corpus 13 / 29 TreeTagger (Schmid, 1999) Corpus: 1.8 million articles of the annotated German Wikipedia Korpus of 2015 Tokenizer: a modifjed version of the tool from Dipper (2016); lemmatizer: Idea: For splits of unknown compounds, each immediate constituent should be found within the context at least somewhere inside a large corpus. For derivatives, this holds only for hypothetical constituents which are free morphs or lexemes. Contexts: the texts of a corpus in which the respective analyzed word form occurs. (Margaretha and Lüngen, 2014) For each text containing the input word form W wf , the document frequencies ( df 1 ...df m ) of the free hypothetical immediate constituents ( c wf ,s, 1 ...c wf ,s,n ) are being retrieved and summarized. This yields a text frequency score ( S wf ,s,t ) for each text and split of n constituents. n � S wf ,s,t = df i c =1 Of all morphological analyses for W wf , the one with the largest score is processed for the storage. A missing hypothetical constituent inside a text containing W wf

  8. Combining Data-Intense Methods with Contextual Retrieval Text indices: for the tokenized and lemmatized forms. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 compensated by the frequencies of the other constituents of the split sequence. leads to a document frequency of 0 for this constituent, which can be (1) Contextual Search in Wikipedia Corpus 13 / 29 TreeTagger (Schmid, 1999) Corpus: 1.8 million articles of the annotated German Wikipedia Korpus of 2015 Tokenizer: a modifjed version of the tool from Dipper (2016); lemmatizer: Idea: For splits of unknown compounds, each immediate constituent should be found within the context at least somewhere inside a large corpus. For derivatives, this holds only for hypothetical constituents which are free morphs or lexemes. Contexts: the texts of a corpus in which the respective analyzed word form occurs. (Margaretha and Lüngen, 2014) For each text containing the input word form W wf , the document frequencies ( df 1 ...df m ) of the free hypothetical immediate constituents ( c wf ,s, 1 ...c wf ,s,n ) are being retrieved and summarized. This yields a text frequency score ( S wf ,s,t ) for each text and split of n constituents. n � S wf ,s,t = df i c =1 Of all morphological analyses for W wf , the one with the largest score is processed for the storage. A missing hypothetical constituent inside a text containing W wf

  9. Combining Data-Intense Methods with Contextual Retrieval Text indices: for the tokenized and lemmatized forms. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 compensated by the frequencies of the other constituents of the split sequence. leads to a document frequency of 0 for this constituent, which can be (1) Contextual Search in Wikipedia Corpus 13 / 29 TreeTagger (Schmid, 1999) Corpus: 1.8 million articles of the annotated German Wikipedia Korpus of 2015 Tokenizer: a modifjed version of the tool from Dipper (2016); lemmatizer: Idea: For splits of unknown compounds, each immediate constituent should be found within the context at least somewhere inside a large corpus. For derivatives, this holds only for hypothetical constituents which are free morphs or lexemes. Contexts: the texts of a corpus in which the respective analyzed word form occurs. (Margaretha and Lüngen, 2014) For each text containing the input word form W wf , the document frequencies ( df 1 ...df m ) of the free hypothetical immediate constituents ( c wf ,s, 1 ...c wf ,s,n ) are being retrieved and summarized. This yields a text frequency score ( S wf ,s,t ) for each text and split of n constituents. n � S wf ,s,t = df i c =1 Of all morphological analyses for W wf , the one with the largest score is processed for the storage. A missing hypothetical constituent inside a text containing W wf

  10. Combining Data-Intense Methods with Contextual Retrieval Text indices: for the tokenized and lemmatized forms. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 compensated by the frequencies of the other constituents of the split sequence. leads to a document frequency of 0 for this constituent, which can be (1) Contextual Search in Wikipedia Corpus 13 / 29 TreeTagger (Schmid, 1999) Corpus: 1.8 million articles of the annotated German Wikipedia Korpus of 2015 Tokenizer: a modifjed version of the tool from Dipper (2016); lemmatizer: Idea: For splits of unknown compounds, each immediate constituent should be found within the context at least somewhere inside a large corpus. For derivatives, this holds only for hypothetical constituents which are free morphs or lexemes. Contexts: the texts of a corpus in which the respective analyzed word form occurs. (Margaretha and Lüngen, 2014) For each text containing the input word form W wf , the document frequencies ( df 1 ...df m ) of the free hypothetical immediate constituents ( c wf ,s, 1 ...c wf ,s,n ) are being retrieved and summarized. This yields a text frequency score ( S wf ,s,t ) for each text and split of n constituents. n � S wf ,s,t = df i c =1 Of all morphological analyses for W wf , the one with the largest score is processed for the storage. A missing hypothetical constituent inside a text containing W wf

  11. Combining Data-Intense Methods with Contextual Retrieval Recheck simple Contextual Search in Wikipedia Corpus NN V NNSUFF NNSUFF no Build all combinations Filter out implausibles Contextual search in Wikipedia corpus analyses Moremorph Frequencies in Wikipedia corpus Weighting by word lengths Analyzable? yes no Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Chefredakteurin Chef redakt eur in SMOR & Lexical DBs Wordlists Bambussieb preußisch-europäisch Hybrid Word Analyzer Morphological Trees (*Aufwand* (*aufwenden* New splits Lexemes Monomorphemic Found in bases? Results : yes subanalyses Check for data- 14 / 29 Arbeitsaufwand (*Arbeit* arbeiten) | s | (*Aufwand* (*aufwenden* auf | wenden)) Arbeitsaufwand Chefredakteurin Chefredakteurin (*Chefredakteur* Chef | (*Redakteur* redakt | eur)) | in Bambus | Sieb (*Arbeit* arbeiten) | s | auf | wenden)) preußisch | - | europäisch Chefredakteur | in Chefredakteur | in ♯ Bambussieb ( ♯ preußisch | - | Europa | isch) Bambus | Sieb

  12. Combining Data-Intense Methods with Contextual Retrieval Recheck simple Contextual Search in Wikipedia Corpus NN V NNSUFF NNSUFF no Build all combinations Filter out implausibles Contextual search in Wikipedia corpus analyses Moremorph Frequencies in Wikipedia corpus Weighting by word lengths Analyzable? yes no Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Chefredakteurin Chef redakt eur in SMOR & Lexical DBs Wordlists Bambussieb preußisch-europäisch Hybrid Word Analyzer Morphological Trees (*Aufwand* (*aufwenden* New splits Lexemes Monomorphemic Found in bases? Results : yes subanalyses Check for data- 15 / 29 Arbeitsaufwand (*Arbeit* arbeiten) | s | (*Aufwand* (*aufwenden* auf | wenden)) Arbeitsaufwand Chefredakteurin Chefredakteurin (*Chefredakteur* Chef | (*Redakteur* redakt | eur)) | in Bambus | Sieb (*Arbeit* arbeiten) | s | auf | wenden)) preußisch | - | europäisch Chefredakteur | in Chefredakteur | in ♯ Bambussieb ( ♯ preußisch | - | Europa | isch) Bambus | Sieb

  13. Combining Data-Intense Methods with Contextual Retrieval Investigations on the lengths of German morphs show that German September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 but b. hypothetical splits with more than one constituent do exist. contextual search found only splits comprising just one constituent Check: all word forms with more than 8 characters if a. the proportional and slightly larger (Krott, 1996). (Menzerath, 1954; Gerlach, 1982). The number of graphemes is simplex lexemes rarely possess more than 7 phonemes (98.41%) Bambussieb Morphological Segmentation based on Corpus Frequencies a. [[NN],[NN]] Bambussieb ‘bamboo screen’ (3) a double check for longer word forms is advisable The corpus itself is considered as a context in the widest sense if Corpus Frequencies 16 / 29 no text contains the word form W wf Bambus | Sieb b. ♯ [[NN, NN]]

  14. Combining Data-Intense Methods with Contextual Retrieval Recheck simple Morphological Segmentation based on Corpus Frequencies NN V NNSUFF NNSUFF no Build all combinations Filter out implausibles Contextual search in Wikipedia corpus analyses Moremorph Frequencies in Wikipedia corpus Weighting by word lengths Analyzable? yes no Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Chefredakteurin Chef redakt eur in SMOR & Lexemes Wordlists Arbeitsaufwand Chefredakteurin Bambussieb preußisch-europäisch Hybrid Word Analyzer Morphological Trees (*Aufwand* (*aufwenden* New splits Monomorphemic 17 / 29 yes Lexical DBs Found in data- bases? Results : Check for subanalyses Arbeitsaufwand (*Arbeit* arbeiten) | s | (*Aufwand* (*aufwenden* auf | wenden)) Chefredakteurin (*Chefredakteur* Chef | (*Redakteur* redakt | eur)) | in Bambus | Sieb (*Arbeit* arbeiten) | s | auf | wenden)) preußisch | - | europäisch Bambus | Sieb Chefredakteur | in Chefredakteur | in ♯ Bambussieb ( ♯ preußisch | - | Europa | isch) Bambus | Sieb

  15. Combining Data-Intense Methods with Contextual Retrieval The functional dependency between morph/lexeme frequency and September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 (2) the document frequencies (2). The Relation between Length and Frequency other factors such as the age of words and lexicon size. length is mutual (Köhler (1986), Krott (2004)) and infmuenced by 18 / 29 a. b. The frequency-based weighting has a bias towards constructions with small constituents. (4) c. ♯ Figur | Kombi | Nation ‘fjgure | combi (short form of combination) | nation’ Figur | Kombination ‘fjgure | combination’ Figur | (*Kombination* kombin | ation) For each constituent with a length of l characters, the frequency of its word length class L l is used as an inverse proportional factor for n df i � WeightedS wf ,s,t = f req ( L l ( c ) ) c =1

  16. Combining Data-Intense Methods with Contextual Retrieval The functional dependency between morph/lexeme frequency and September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 (2) the document frequencies (2). The Relation between Length and Frequency other factors such as the age of words and lexicon size. length is mutual (Köhler (1986), Krott (2004)) and infmuenced by 18 / 29 a. b. The frequency-based weighting has a bias towards constructions with small constituents. (4) c. ♯ Figur | Kombi | Nation ‘fjgure | combi (short form of combination) | nation’ Figur | Kombination ‘fjgure | combination’ Figur | (*Kombination* kombin | ation) For each constituent with a length of l characters, the frequency of its word length class L l is used as an inverse proportional factor for n df i � WeightedS wf ,s,t = f req ( L l ( c ) ) c =1

  17. Combining Data-Intense Methods with Contextual Retrieval The functional dependency between morph/lexeme frequency and September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 (2) the document frequencies (2). The Relation between Length and Frequency other factors such as the age of words and lexicon size. length is mutual (Köhler (1986), Krott (2004)) and infmuenced by 18 / 29 a. b. The frequency-based weighting has a bias towards constructions with small constituents. (4) c. ♯ Figur | Kombi | Nation ‘fjgure | combi (short form of combination) | nation’ Figur | Kombination ‘fjgure | combination’ Figur | (*Kombination* kombin | ation) For each constituent with a length of l characters, the frequency of its word length class L l is used as an inverse proportional factor for n df i � WeightedS wf ,s,t = f req ( L l ( c ) ) c =1

  18. Combining Data-Intense Methods with Contextual Retrieval Recheck simple The Relation between Length and Frequency NN V NNSUFF NNSUFF no Build all combinations Filter out implausibles Contextual search in Wikipedia corpus analyses Moremorph Frequencies in Wikipedia corpus Weighting by word lengths Analyzable? yes no Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Chefredakteurin Chef redakt eur in SMOR & Lexemes Wordlists Arbeitsaufwand Chefredakteurin Hybrid Word Analyzer Morphological Trees (*Aufwand* (*aufwenden* New splits Monomorphemic 19 / 29 Lexical DBs bases? Results : yes subanalyses Check for data- Found in Arbeitsaufwand (*Arbeit* arbeiten) | s | (*Aufwand* (*aufwenden* auf | wenden)) Chefredakteurin (*Chefredakteur* Chef | (*Redakteur* redakt | eur)) | in Bambussieb preußisch-europäisch preußisch | - | (*europäisch* Europa|isch) (*Arbeit* arbeiten) | s | auf | wenden)) preußisch | - | europäisch Bambus | Sieb Chefredakteur | in Chefredakteur | in ♯ Bambussieb ( ♯ preußisch | - | Europa | isch) Bambus | Sieb

  19. Evaluation Outline 1 Introduction 2 Combining Data-Intense Methods with Contextual Retrieval 3 Evaluation Test Data Results of Hybrid Word Analyzing 4 Conclusions and Future Work Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 20 / 29

  20. Evaluation monomorphemic words. Coverage of 55.99% with an accuracy of September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 sample of 1,006 word forms and Moremorph with a coverage of 100%. The remaining 44.01% of all lemma types were processed by SMOR nearly 100% due to the quality of the CELEX and GermaNet data. 15,622 of these lemma types are inside the databases of trees or Test Data 38,337 word-form types, and 27,902 lemma types. 276 texts with 5,202 paragraphs, 16,046 sentences and 260,114 tokens Tokenization: enlarged and costumized tokenizer by Dipper (2016) consumption and aviation. Kupietz et al. 2010), an in-fmight magazine with articles on traveling, DeReKo-2016-I (Institut für Deutsche Sprache 2016) corpus (see Corpus: Korpus Magazin Lufthansa Bordbuch (MLD) , part of the 21 / 29

  21. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  22. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  23. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  24. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  25. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  26. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  27. Evaluation (5) Results of Hybrid Word Analyzing + Recheck + Weighting 93.34% 2.58% 2.78% 1.29% e. frequent homograph: 2.68% f. correction by word-length weighting: rollen|(*Vorgang* (*vorgehen* vor|gehen)) ‘to roll|(*procedure* (*to proceed* pro|ceed))’ 5,696 new entries for monomorphemic lexemes; 8,448 for the new splits. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2.88% 1.99% 92.44% + Recheck correct analysis (all levels) no analysis (partially) fmat analysis wrong analysis DBs 22 / 29 text + Corpus DBs + Con- Look-up 87.77% 7.45% 3.48% 1.29% ≈ 55.99% a. adjective vs. participle: folgend ‘following’, gewandt ‘turned v , skillful adj ’ b. constituents not in context: Metallkäfjg ‘metal cage’, Tierärztin ‘vet f em ’ c. fmat analyses: Roll | vor | Gang ‘ ♯ ? (to roll | prefjx, before | gait), rolling procedure’ d. analysis from GermaNet: ♯ ? (Land | Nahme) ‘(land | ”take”), settlement’ ♯ (Parlament|(*arisch* ar|isch)) ‘(parliament|(*Aryan* Ar|ian), parliamentary’

  28. Conclusions and Future Work Outline 1 Introduction 2 Combining Data-Intense Methods with Contextual Retrieval 3 Evaluation 4 Conclusions and Future Work Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 23 / 29

  29. Conclusions and Future Work An Hybrid Approach for Deep-Level Morphological Analysis Starting points: a. morphological trees database b. fmat structures from a morphological segmentation tool. All plausible combinations of the immediate constituents were evaluated by look-ups in textual environments of a large corpus or inside the set of all types as a back-ofg strategy. Biases towards small constituents with high frequencies on the one side and unsplit words on the other were tackled by insights from investigations in quantitative linguistics. The combination of the methods lead to an accuracy of 93% for complex structures and 98.7% for acceptable output. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 24 / 29

  30. Conclusions and Future Work An Hybrid Approach for Deep-Level Morphological Analysis Starting points: a. morphological trees database b. fmat structures from a morphological segmentation tool. All plausible combinations of the immediate constituents were evaluated by look-ups in textual environments of a large corpus or inside the set of all types as a back-ofg strategy. Biases towards small constituents with high frequencies on the one side and unsplit words on the other were tackled by insights from investigations in quantitative linguistics. The combination of the methods lead to an accuracy of 93% for complex structures and 98.7% for acceptable output. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 24 / 29

  31. Conclusions and Future Work An Hybrid Approach for Deep-Level Morphological Analysis Starting points: a. morphological trees database b. fmat structures from a morphological segmentation tool. All plausible combinations of the immediate constituents were evaluated by look-ups in textual environments of a large corpus or inside the set of all types as a back-ofg strategy. Biases towards small constituents with high frequencies on the one side and unsplit words on the other were tackled by insights from investigations in quantitative linguistics. The combination of the methods lead to an accuracy of 93% for complex structures and 98.7% for acceptable output. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 24 / 29

  32. Conclusions and Future Work An Hybrid Approach for Deep-Level Morphological Analysis Starting points: a. morphological trees database b. fmat structures from a morphological segmentation tool. All plausible combinations of the immediate constituents were evaluated by look-ups in textual environments of a large corpus or inside the set of all types as a back-ofg strategy. Biases towards small constituents with high frequencies on the one side and unsplit words on the other were tackled by insights from investigations in quantitative linguistics. The combination of the methods lead to an accuracy of 93% for complex structures and 98.7% for acceptable output. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 24 / 29

  33. Conclusions and Future Work Future Work For improvement, there are two directions: using larger corpora, to possibly obtain a better fjt of the wordlength-frequency relationship. On the other hand, inhomogeneous data can blur models. Therefore, analyzing words text by text could help to achieve larger contextual dependency and to fjnd morphological structures fjtting to the direct environment. This would result in difgerent structures for orthographical words according to their contexts. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 25 / 29

  34. 26 / 29 Prefjx September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 keit Suffjx sam Suffjx ‘to notice’ merken V auf ‘to attend to’ Daueraufmerksamkeit aufmerken V ‘attentive’ aufmerksam Adj ‘attention’ Aufmerksamkeit N ‘endurance’ Dauer N Thank you for your ‘permanent | attention’

  35. References I in GermaNet. In Proceedings of the International Conference Recent Advances in Natural September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 https://doi.org/10.1080/09296179608590061. Journal of Quantitative Linguistics 3(1):29–37. Andrea Krott. 1996. Some remarks on the relation between word length and morpheme length. Quantitative Linguistics 31. Studienverlag Dr. N. Brockmeyer, Bochum. Reinhard Köhler. 1986. Zur linguistischen Synergetik: Struktur und Dynamik der Lexik . pages 420–426. http://www.aclweb.org/anthology/R11-1058. Language Processing, Hissar, Bulgaria, 2011 . Association for Computational Linguistics, Verena Henrich and Erhard Hinrichs. 2011. Determining Immediate Constituents of Compounds Harald Baayen, Richard Piepenbrock, and Léon Gulikers. 1995. The CELEX lexical database http://www.aclweb.org/anthology/W97-0802. Semantic Resources for NLP Applications . pages 9–15. Proceedings of ACL Workshop Automatic Information Extraction and Building of Lexical Birgit Hamp and Helmut Feldweg. 1997. GermaNet - a Lexical-Semantic Net for German. In Quantitative Linguistics 14, pages 95–102. Morphologie. In W. Lehfeldt and U. Strauss, editors, Glottometrika 4 , Brockmeyer, Rainer Gerlach. 1982. Zur Überprüfung des Menzerathschen Gesetzes im Bereich der https://www.linguistics.rub.de/~dipper/resources/tokenizer.html. Stefanie Dipper. 2016. Tokenizer for German. (CD-ROM). 27 / 29

  36. References II Paul Menzerath. 1954. Die Architektonik des deutschen Wortschatzes . Phonetische Studien. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 https://doi.org/10.1007/978-94-017-2390-9_2. Springer Netherlands, Dordrecht, pages 13–25. and David Yarowsky, editors, Natural Language Processing Using Very Large Corpora , In Susan Armstrong, Kenneth Church, Pierre Isabelle, Sandra Manzi, Evelyne Tzoukermann, Helmut Schmid. 1999. Improvements in Part-of-Speech Tagging with an Application to German. Dümmler, Bonn ; Hannover ; Stuttgart. http://www.jlcl.org/2014_Heft2/3MargarethaLuengen.pdf. Andrea Krott. 2004. Ein funktionalanalytisches Modell der Wortbildung [A functional analytical http://nbn-resolving.de/urn:nbn:de:bsz:mh39-33306, challenges at the interface between computational and corpus linguistics 29(2):59 – 82. issue on building and annotating corpora of computer-mediated communication. Issues and and discussions. Journal of Language Technology and Computational Linguistics. Special Eliza Margaretha and Harald Lüngen. 2014. Building linguistic corpora from wikipedia articles http://ubt.opus.hbz-nrw.de/volltexte/2004/279/pdf/04_krott.pdf. Universität Trier, Trier, pages 75–126. Quantitative and System-theoretical Linguistics] , Elektronische Hochschulschriften an der zur Quantitativen und Systemtheoretischen Linguistik [Corpus-linguistic Investigations of model of word formation]. In Reinhard Köhler, editor, Korpuslinguistische Untersuchungen 28 / 29

  37. References III Science. September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 https://www.aclweb.org/anthology/L18-1613. (LREC 2018), Miyazaki, Japan . European Language Resources Association (ELRA). Proceedings of the Eleventh International Conference on Language Resources and Evaluation Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène from a Linguistic Database. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Petra Steiner and Josef Ruppenhofer. 2018. Building a Morphological Treebank for German Issues in Data Science, March 5-8, 2019, Zanjan, Iran . Springer, Lecture Notes in Computer Helmut Schmid, Arne Fitschen, and Ulrich Heid. 2004. SMOR: A German Computational Morphological Parsing. In Proceedings of The International Conference on Contemporary Petra Steiner and Reinhard Rapp. in press. Building and Exploiting Lexical Databases for https://aclweb.org/anthology/W17-7619. Linguistic Theories, January 23-24, 2018, Prague, Czech Republic . pages 146–160. Two Resources. In Proceedings of the 16th International Workshop on Treebanks and Petra Steiner. 2017. Merging the Trees - Building a Morphological Treebank for German from http://www.aclweb.org/anthology/L04-1275. 2004, Lisbon, Portugal . European Language Resources Association (ELRA). International Conference on Language Resources and Evaluation, LREC 2004, May 26-28, Morphology Covering Derivation, Composition and Infmection. In Proceedings of the Fourth 29 / 29

  38. More slides .... Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 1 / 44

  39. Figure 4: length/frequeny of lemmas from MLD corpus Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 2 / 44

  40. Databases Reliable Morphological Data CELEX: standard resource for German lexical data 51,728 entries 38,650 derivatives or compounds, 2,402 conversions core vocabulary outdated format outdated spelling GermaNet: rich vocabulary, complex lexemes segmentation restricted to nominal compounds approx. 68,000 entries of compounds Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 3 / 44

  41. Databases Revised German Morphology - Lemmas yes no Morph-orthographical data Intermediate Results Control output Substitution of morphs check and add revise spelling no Consonant rules Generate substitution rules New German Morphology - Lemmas (GMOL) check and add see ? Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Simple substitution yes Revised CELEX Database Orthographical Dataset Transfer of CELEX to the Modern Standard outdated format, e.g. Abschlu$ (orthographical dataset) German morphology lemmas (GML) German orthography lemmas (GOL) no change Morphological Dataset Change character Dia- critic more than one Substitution with cotext 4 / 44 Abschluss (morphological dataset) �− → Abschluß ‘conclusion’ outdated spelling, e.g. Abschluß ‘conclusion’ �− → Abschluss

  42. Databases Germa- September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 German CELEX- OrthCELEX German CELEX- Refurbished Net (with CELEX) Revised CELEX Database GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 5 / 44

  43. Databases Germa- September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 German CELEX- OrthCELEX German CELEX- Refurbished Net (with CELEX) Revised CELEX Database GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 5 / 44

  44. Databases 207\Abgangszeugnis\4\C\1\Y\Y\Y\Abgang+s+Zeugnis\NxN\N\N\N\ September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Morphological Trees of CELEX ((((ab)[V|.V],(geh)[V])[V])[N],(s)[N|N.N],((zeug)[V],(nis)[N|V.])[N]) [...] 97\Abdrift\0\C\1\Y\Y\Y\ab+drift\xV\N\N\N\ ((ab)[N|.V],((treib)[V])[V])[N]\Y\N\N\N\S3/P3\N Examples CELEX Trees I 6 / 44 ‘leeway - away | to fmoat’ ‘leaving certifjcate - leave | certifjcate’ 605\Abschlussprüfung\C\1\Y\Y\Y\Abschluss + Prüfung\NN\N\N\N\ ((((ab)[V | .V],(schließ)[V])[V])[N],((prüf)[V],(ung)[N | V.])[N]\[...] ‘fjnal exam - conclusion | exam’

  45. Databases V September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Figure 5: Morphological analysis of Abschlussprüfung ‘fjnal exam’ suffjx ung ‘examine’ prüf N Morphological Trees of CELEX ‘close’ schließ V ‘away’ ab N NN CELEX Trees II 7 / 44

  46. Databases N September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Figure 6: Morphological analysis of Abgangszeugnis ‘leaving certifjcate’ suffjx nis ‘to witness’ zeug V ‘interfjx’ Morphological Trees of CELEX s x ‘to go’ geh V ‘away’ ab N NN CELEX Trees III 8 / 44

  47. Databases Germa- September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 German CELEX- OrthCELEX German CELEX- Refurbished Net (with CELEX) Morphological Trees of CELEX GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 9 / 44

  48. Databases Germa- September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 German CELEX- OrthCELEX German CELEX- Refurbished Net (with CELEX) Morphological Trees of CELEX GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 9 / 44

  49. Databases sense=”1” source=”core” namedEntity=”no” artifjcial=”no” September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Morphological Trees from GermaNet 10 / 44 same approach as WordNet (Hamp and Feldweg, 1997) Lexical-semantic database, hierarchically structured in synsets GermaNet < synset id=”s5552” category=”nomen” class=”Artefakt” > < lexUnit id=”l8355” styleMarking=”no” > < orthForm > Werkstück < /orthForm > < compound > < modifjer category=”Nomen” > Werk < /modifjer > < modifjer category=”Verb” > werken < /modifjer > < head > Stück < /head > < /compound > < /lexUnit > < /synset >

  50. Databases sense=”1” source=”core” namedEntity=”no” artifjcial=”no” September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Morphological Trees from GermaNet 10 / 44 same approach as WordNet (Hamp and Feldweg, 1997) Lexical-semantic database, hierarchically structured in synsets GermaNet < synset id=”s5552” category=”nomen” class=”Artefakt” > < lexUnit id=”l8355” styleMarking=”no” > < orthForm > Werkstück < /orthForm > < compound > < modifjer category=”Nomen” > Werk < /modifjer > < modifjer category=”Verb” > werken < /modifjer > < head > Stück < /head > < /compound > < /lexUnit > < /synset >

  51. Databases affjxoids September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Abfahrtszeit ‘departure time’ Morphological Trees from GermaNet add interfjxes (Fugen/fjller letters) by heuristics remove defjcient entries, e.g. with missing parts-of-speech classes or Bodenseeregion ‘Lake of Constance region’) remove proper names, foreign word expressions ( After-Show-Party , 66,059 compounds of which some have ambiguous structures GermaNet compounds (Henrich and Hinrichs, 2011), version 11 with GN Trees I: Compounds of GermaNet 11 / 44 GermaNet: Abfahrt | zeit �− → Abfahrt | s | zeit

  52. Databases Morphological Trees from GermaNet September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 insert insert ((Beitrag s Satz) Sicherung) s Gesetz (Beitrag s Satz) Sicherung Beitrag s Satz Beitragssatz ‘contribution rate safeguarding law’ ‘contribution rate safeguarding’ GN Trees II: Compound structures from GermaNet 12 / 44 1 generate fmat compound entries Beitragssatz : Beitrag | s | Satz ‘contribution rate’ Beitragssatzsicherung : Beitragssatz | Sicherung Beitragssatzsicherungsgesetz : Beitragssatzsicherung | s | Gesetz 2 infer GN complex structure by recursive look-up Beitragssatz | Sicherung Beitragssatzsicherung | s | Gesetz

  53. Databases Morphological Trees from GermaNet September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 insert insert ((Beitrag s Satz) Sicherung) s Gesetz (Beitrag s Satz) Sicherung 12 / 44 Beitragssatz ‘contribution rate safeguarding law’ ‘contribution rate safeguarding’ GN Trees II: Compound structures from GermaNet 1 generate fmat compound entries Beitragssatz : Beitrag | s | Satz ‘contribution rate’ Beitragssatzsicherung : Beitragssatz | Sicherung Beitragssatzsicherungsgesetz : Beitragssatzsicherung | s | Gesetz 2 infer GN complex structure by recursive look-up Beitrag | s | Satz Beitragssatz | Sicherung Beitragssatzsicherung | s | Gesetz

  54. Databases Beitragssatz September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 insert insert ((Beitrag s Satz) Sicherung) s Gesetz Morphological Trees from GermaNet 12 / 44 ‘contribution rate safeguarding’ ‘contribution rate safeguarding law’ GN Trees II: Compound structures from GermaNet 1 generate fmat compound entries Beitragssatz : Beitrag | s | Satz ‘contribution rate’ Beitragssatzsicherung : Beitragssatz | Sicherung Beitragssatzsicherungsgesetz : Beitragssatzsicherung | s | Gesetz 2 infer GN complex structure by recursive look-up Beitrag | s | Satz Beitragssatz | Sicherung (Beitrag | s | Satz) | Sicherung Beitragssatzsicherung | s | Gesetz

  55. Databases Beitragssatz September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 insert insert Morphological Trees from GermaNet 12 / 44 ‘contribution rate safeguarding law’ ‘contribution rate safeguarding’ GN Trees II: Compound structures from GermaNet 1 generate fmat compound entries Beitragssatz : Beitrag | s | Satz ‘contribution rate’ Beitragssatzsicherung : Beitragssatz | Sicherung Beitragssatzsicherungsgesetz : Beitragssatzsicherung | s | Gesetz 2 infer GN complex structure by recursive look-up Beitrag | s | Satz Beitragssatz | Sicherung (Beitrag | s | Satz) | Sicherung Beitragssatzsicherung | s | Gesetz ((Beitrag | s | Satz) | Sicherung) | s | Gesetz

  56. Databases Net September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 (Steiner, 2017) German CELEX- OrthCELEX German CELEX- Refurbished Germa- Morphological Trees from GermaNet (with CELEX) GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 13 / 44

  57. Databases Net September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 (Steiner, 2017) German CELEX- OrthCELEX German CELEX- Refurbished Germa- Morphological Trees from GermaNet (with CELEX) GNextract CELEXextract Trees DB GermaNet DB CELEX Trees Trees DB Morphological Merged Overview of the Data Processing 13 / 44

  58. Combining CELEX and GermaNet infer complex (derivative) structures by recursive look-up September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 approx. 2000 missing segmentations Some mistakes/questionable or missing analyses: Beitragssatzsicherungsgesetz Beitragssatzsicherung Building the Trees Beitragssatz Gesetz ‘law’ Sicherung ‘safeguarding’ Beitrag ‘contribution’ immediate constituents and other information from CELEX GermaNet compound structures Combining GN Trees and CELEX Trees 14 / 44 (*beitragen V * bei x | tragen V ) (*sichern V * sicher A | n x ) | ung x ge x | setzen V Beitrag | s | Satz (Beitrag | s | Satz) | (sichern | ung) ((Beitrag | s | Satz) | (sichern | ung)) | s | (ge | setzen) Restrukturierungsmaßnahmen : Restrukturierung | s | (Maß | Nahme)

  59. Combining CELEX and GermaNet Removing compounds with proper names and/or foreign words as September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Dissimilarity measure for CELEX diachronic analyses Depth of analysis for conversions for CELEX Analysis of conversions for CELEX constituents for GN Transfering the GN annotation scheme to CELEX scheme Building the Trees Addition of fjller letters for GN splits on the same level) Parts of speech for the constructs and/or the smallest constituents Depth of analysis for compounds Optional parameters: Formats of Output 15 / 44 Choice of the output format (parentheses or a notation with | for the

  60. Combining CELEX and GermaNet gleich ‘to adjust’ x aus V gleichen ‘to equal’ Adj ‘equal’ V x en Figure 7: Merged morphological analysis of Währungsausgleichsfonds ‘currency adjustment fund’ Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 ausgleichen Ausgleich Merged German Morphological Trees ‘currency’ Example Währungsausgleichsfonds N Währungsausgleich ‘currency adjustment’ N Währung x ‘fund’ s N Ausgleich ‘adjustment’ x s N Fonds 16 / 44

  61. Combining CELEX and GermaNet gleich ‘to adjust’ x aus V gleichen ‘to equal’ Adj ‘equal’ V x en Figure 7: Merged morphological analysis of Währungsausgleichsfonds ‘currency adjustment fund’ Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 ausgleichen Ausgleich Merged German Morphological Trees ‘currency’ Example Währungsausgleichsfonds N Währungsausgleich ‘currency adjustment’ N Währung x ‘fund’ s N Ausgleich ‘adjustment’ x s N Fonds 16 / 44

  62. Combining CELEX and GermaNet gleich ‘to adjust’ x aus V gleichen ‘to equal’ Adj ‘equal’ V x en Figure 7: Merged morphological analysis of Währungsausgleichsfonds ‘currency adjustment fund’ Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 ausgleichen Ausgleich Merged German Morphological Trees ‘currency’ Example Währungsausgleichsfonds N Währungsausgleich ‘currency adjustment’ N Währung x ‘fund’ s N Ausgleich ‘adjustment’ x s N Fonds 16 / 44

  63. Combining CELEX and GermaNet s ‘to equal’ Adj gleich ‘equal’ x en x N V Fonds ‘fund’ Figure 8: Merged morphological analysis of Währungsausgleichsfonds ‘currency adjustment fund’ Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 gleichen aus Merged German Morphological Trees ‘currency’ Example Währungsausgleichsfonds N Währungsausgleich ‘currency adjustment’ N Währung x x s N Ausgleich ‘adjustment’ V ausgleichen ‘to adjust’ 17 / 44

  64. Combining CELEX and GermaNet Merged German Morphological Trees September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 conversions; c: fmat representation of the immediate constituent. c. Abgangszeugnis Abgang_N|s_x|Zeugnis_N (s_x)(*Zeugnis_N* (zeugen_V)(nis_x)) b. Abgangszeugnis (*Abgang_N* (*abgehen_V* (ab_x)(gehen_V))) |s_x|*Zeugnis_N* (zeugen_V|nis_x) a. Abgangszeugnis (*Abgang_N* (*abgehen_V* ab_x|gehen_V)) c. Abdrift ab_x|driften_V b. Abdrift (ab_x)(*driften_V* treiben_V) a. Abdrift ab_x|(driften_V) c. Abschlussprüfung Abschluss_N|Prüfung_N (schließen_V)))(*Prüfung_N* (prüfen_V)(ung_x)) b. Abschlussprüfung (*Abschluss_N* (*abschließen_V* (ab_x) ab_x|schließen_V))|(*Prüfung_N* prüfen_V|ung_x) a. Abschlussprüfung (*Abschluss_N* (*abschließen_V* Small Examples of List Representations 18 / 44 a: | notation, threshold 0.5; b: parenthesis notation and no restrictions on diachronic

  65. Combining CELEX and GermaNet 104,424 September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Table 1: Databases of German word trees 112,086 n/a 68,171 plus simplex words merged with CELEX 100,986 n/a 68,171 merged with CELEX 40,097 Merged German Morphological Trees 68,163 deep-level 100,095 40,097 67,452 fmat German Trees CELEX entries GN entries Structures set to 0.5. Double entries were removed. words and 2 for conversions. The Levenshtein dissimilarity threshold was The parameters for the deep-level analyses are 6 for the levels of complex Merged GermanTrees 19 / 44

  66. Combining Databases and Segmenters by Hybrid Word Analysis Zeugnis<NN> (*abgehen_V* (ab_x) (gehen_V))) (s_x) (*Zeugnis_N* (zeugen_V) (nis_x)) SMOR/Moremorph Abgang<N>s<FL> Morphy German SUB NOM SIN NEU KMP Abgang/Zeugnis Figure 9: Morphological trees database and two difgerent word segmenters as alternative methods for word splitting Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 (*Abgang_N* CELEX- One Complex Database, Two Segmenters simplex words Wordlists: Abgangszeugnis Hybrid Word Splitter Morphological Trees DB CELEX Trees & GermaNet Trees OrthCELEX CELEXextract GNextract (withCELEX) GermaNet Refurbished CELEX- German 20 / 44

  67. Evaluation: Testcorpus and Recall + Moremorphs 60.59% + Morphy 21,953 74.89% 241,117 92.73% 27,907 49.29% 95.20% 256,903 98.80% Table 2: Recall of Tree DBs (Steiner and Rapp, in press) Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 157,535 14,446 Coverage of the Lemma Forms lemma Corpus: Korpus Magazin Lufthansa Bordbuch (MLD) , part of the DeReKo-2016-I (Institut für Deutsche Sprache 2016) corpus (see Kupietz et al. 2010), an in-fmight magazine with articles on traveling, consumption and aviation. Tokenization: enlarged and costumized tokenizer by Dipper (2016) 276 texts with 5,202 paragraphs, 16,046 sentences and 260,115 tokens types MergedDB + simplex recall lemmas in text recall corpus size 29,313 260,014 21 / 44

  68. Evaluation: Testcorpus and Recall + Moremorphs 60.59% + Morphy 21,953 74.89% 241,117 92.73% 27,907 49.29% 95.20% 256,903 98.80% Table 2: Recall of Tree DBs (Steiner and Rapp, in press) Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 157,535 14,446 Coverage of the Lemma Forms lemma Corpus: Korpus Magazin Lufthansa Bordbuch (MLD) , part of the DeReKo-2016-I (Institut für Deutsche Sprache 2016) corpus (see Kupietz et al. 2010), an in-fmight magazine with articles on traveling, consumption and aviation. Tokenization: enlarged and costumized tokenizer by Dipper (2016) 276 texts with 5,202 paragraphs, 16,046 sentences and 260,115 tokens types MergedDB + simplex recall lemmas in text recall corpus size 29,313 260,014 21 / 44

  69. Evaluation: Testcorpus and Recall + Moremorphs 60.59% + Morphy 21,953 74.89% 241,117 92.73% 27,907 49.29% 95.20% 256,903 98.80% Table 2: Recall of Tree DBs (Steiner and Rapp, in press) Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 157,535 14,446 Coverage of the Lemma Forms lemma Corpus: Korpus Magazin Lufthansa Bordbuch (MLD) , part of the DeReKo-2016-I (Institut für Deutsche Sprache 2016) corpus (see Kupietz et al. 2010), an in-fmight magazine with articles on traveling, consumption and aviation. Tokenization: enlarged and costumized tokenizer by Dipper (2016) 276 texts with 5,202 paragraphs, 16,046 sentences and 260,115 tokens types MergedDB + simplex recall lemmas in text recall corpus size 29,313 260,014 21 / 44

  70. Conclusion Conclusion 100,986 merged trees Currently the biggest available data resource of its kind Text coverage of 60.59% Combined with Morphy: 92.73% Combined with SMOR: 98.80% Downloads without data: https://github.com/petrasteiner/morphology The authors were partially supported by the German Research Foundation (DFG) under grant RU 1873/2-1 and by a Marie Curie Career Integration Grant within the 7th European Community Framework Programme. Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 22 / 44

  71. Conclusion Abschluss (morphological dataset) September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 outdated spelling, e.g. Revised CELEX Database 23 / 44 Abschlu$ (orthographical dataset) outdated format, e.g. manually annotated multi-tiered word structures (Baayen et al., 1995) combined with information on word-formation types and frequencies Dutch, English, and German lexical information The Lexical Database CELEX �− → Abschluß (modern format) ‘conclusion’ Abschluß ‘conclusion’ �− → Abschluss (modern spelling).

  72. Conclusion Consonant rules Intermediate Results Control output Substitution of morphs check and add Revised German Morphology - Lemmas revise spelling Generate substitution rules no New German Morphology - Lemmas (GMOL) check and add see ? Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Morph-orthographical data yes Revised CELEX Database Dia- Transfer of CELEX to the Modern Standard German morphology lemmas (GML) German orthography lemmas (GOL) Morphological Dataset Orthographical Dataset Change character critic Simple substitution more than one Substitution with cotext no change yes no 24 / 44

  73. Conclusion Revised CELEX Database CELEX Revision - Facts and Figures 51,728 entries 10,106 entries with diacritics 576 entries with updated spelling 38,683 complex entries (morphological deep-level analyses) Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 25 / 44

  74. Conclusion schließ September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 suffjx ung NSUFF ‘examine’ prüf V N ‘close’ V Revised CELEX Database ‘away’ ab VPREF N NN Abschlusspruefung (‘fjnal exam’) Example (Morphological analysis of Abschlußprüfung ‘fjnal exam’) needs only a few repairs of missing constituents or wrong analyses manually annotated multi-tiered word structures (Baayen et al., 1995) The Lexical Database CELEX: Morphological Structures 26 / 44 Abschluss + Pruefung ((((ab)[V | .V],(schliess)[V])[V])[N], ((pruef)[V],(ung)[N | V.])[N])

  75. Conclusion Example: the stem of the derived form treib and its component September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 Plus a small list of exceptions. Steiner and Ruppenhofer (2018) (4) that the analysis will stop for a threshold at 0.8 or below. CELEX Trees IV: Restriction of Diachronic Information driften are reduced to the smaller size (5): drift and treib . (4) shows (3) values is compared to a threshold t as in (3): and ss , uppercase characters to lowercase. Then the quotient of both (LD) of these. Special characters such as ä or ß are transformed to a 27 / 44 Cut two forms f 1 , f 2 with length l 1 and l 2 to the strings s 1 , s 2 of the smaller length ( min ( l 1 ,l 2 )) and calculate the Levensthein distance LD ( s 1 ,s 2 ) min ( l 1 ,l 2 ) < t LD ( drift , treib ) = 4 min ( l 1 ,l 2 ) 5

  76. Conclusion analysedeeper subpart then if part is simplex or depth of analysis reached sub analysedeepercelex part (parameters and level) end end return result of analysedeeper subpart foreach subpart of part do return linguistic information and part depth of analysis++; else end return result of analysedeepercelex analysedeepercelex part with parameters and depth; depth of analysis++; else if constituent not found in GN data then retrieve linguistic information/PoS as required; end return linguistic information and part return result of analysedeepercelex September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 end end end subpart else else end return subpart skip deeper analysis; analysedeepercelex subpart is dissimilar then if levenshtein threshold and analysedeepercelex subpart foreach subpart of part do end retrieve linguistic information/PoS as required; Algorithm 1: Building a merged morphological treebank foreach constituent of entry do end constituent return linguistic information and required; retrieve linguistic information/PoS as if depth of analysis reached then if entry is a compound then then forall entries of GN fmat compounds do add CELEX data to the knowledge base output; information, levenshtein threshold, parts of speech, style of initialization of parameters: depth of analysis, linguistic Output: A DB of Morphological Trees Input: CELEX-German revised, GN fmat compounds else if constituent not found in GN data depth of analysis++; then return result of analysedeeper if part is simplex or depth of analysis reached sub analysedeeper part (parameters and level) end end end end end parameters and depth; analysedeepercelex part with analysedeeper part with depth of analysis++; foreach part of constituent do else end return result of analysedeepercelex parameters and depth; 28 / 44

  77. Conclusion Confmicts September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 ‘(*to work_V* (work_N)(en(suffjx))(piece_N)’ (*werken_V* (Werk_N)(en_x))(Stück_N) Werkstück Glaswerkstück 29 / 44 Werkstück Glaswerkstück Werkstück Examples (GermaNet vs. CELEX) Conversion or ambiguity? Werk | Stück ‘work(noun) | piece’ werken | Stück ‘to work | piece’ Glas | (Werk | Stück) ‘glass | work(noun) | piece’ Glas | (werken | Stück) ‘glass | to work | piece’

  78. Conclusion (afro_x)(amerikanisch_A) September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 ‘(to measure_take_V) measure’ maßnehmen_V ‘(measure_n)(taking_N) measure’ (Maß_N)(Nahme_N) Maßnahme ‘(afro_x)(American_A)’ afroamerikanisch Confmicts ‘(afro_R)(Asian_N)’ (afro_R)(Asiatisch_N) afroasiatisch ‘(away_x)(water_N) waste water’ (ab_x)(Wasser_N) ‘(away_P)(water_N) waste water’ (ab_P)(Wasser_N) Abwasser Examples (GermaNet vs. CELEX) Compounding or Prefjxation/Conversion? 30 / 44

  79. Conclusion n I I pronoun Pronomen O O abbreviation Abkürzung X X word group Wortgruppe n interjection root/confjx Konfjx R R fjller letters, affjxes - x x Table 3: Mapping of two morphological tagsets Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 Interjektion D Confmicts A Mapping the Morphological Tagsets Part of Speech/morph type GN CELEX GN Trees noun nomen, Nomen N N adjective Adjektiv A adverb D Adverb B B preposition Präposition P P verb Verb, verben V V article Artikel 31 / 44

  80. Conclusion Outlook September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 German CELEX- OrthCELEX German CELEX- Refurbished GermaNet (withCELEX) GNextract CELEXextract Trees GermaNet CELEX Trees Outlook Wordlists: Abgangszeugnis Moremorph Abgang<N>s<FL> Zeugnis<NN> Contextual Word Splitter Morphological Trees DB Indices Buildindex Wikipedia Corpus 32 / 44 (*Abgang N * (*abgehen V * (ab x ) (gehen V ))) (s x ) (*Zeugnis N * (zeugen V ) (nis x ))

  81. Frequencies aus t 974 keit 394 ation 983 423 ab en 1141 un 455 los 1215 ier 382 896 ent 374 September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 as found in the Mannheim Corpus . all immediate constituents within the lemmas with their frequencies (21,406 entries) all immediate constituents with their frequencies within the lemmas in unter 836 isch 378 zu 845 an 381 475 1236 Frequencies from CELEX 3066 681 auf 2531 ig 750 ge er 2327 755 ein 3588 ung all morphs with their frequencies within the lemmas (13,419 entries) Three datasets with frequency information were extracted from CELEX: Morphs and immediate constituents I s über be n 475 heit 1273 lich 485 bar 1581 517 630 vor 1694 ver 557 um 2120 e 33 / 44

  82. Frequencies heit be 631 fahren 1537 ver 622 Bau 1452 en 620 1197 stellen schaft 577 bauen 913 es 569 Arbeit Petra Steiner @ DeriMo2019 Fine-Grained Morphological Analyses September 19, 2019 1564 634 Frequencies from GermaNet + CELEX 5939 Morphs and immediate constituents II Preliminary results: smallest parts of GN trees. 11905 s 771 al 8961 ung 730 Land n ge 728 ation 5339 e 721 ion 5324 er 715 Zeit 2198 34 / 44

  83. Frequencies Frequencies from GermaNet + CELEX September 19, 2019 Fine-Grained Morphological Analyses Petra Steiner @ DeriMo2019 ‘drill’ Bohrmaschine ‘*class hit’ Klassenschlag ‘premium’ Ober ‘hammer drill’ Schlagbohrmaschine ‘premium class’ Oberklasse Oberklassenschlagbohrmaschine many combinatorially possible analyses (machine)’ ‘Premium class compact hammer drill Kompaktschlagbohrmaschine Oberklasse- and derivation most common are compounding word formation language with complex processes of Characteristics of German Word-Formation Version 2 35 / 44 ♯ Oberklassenschlagbohrmaschine

Recommend


More recommend