Detecting Errors in Semantic Annotation Argument identification - PowerPoint PPT Presentation

Detecting Errors in Semantic Annotation Introduction & Motivation Background Detecting semantic annotation errors Argument labeling variation Detecting Errors in Semantic Annotation Argument identification variation Heuristics for disambiguating strings Evaluation Markus Dickinson and Chong Min Lee Results Insights Indiana University and Georgetown University Summary & Outlook LREC 2008, Marrakech, Morocco References 1 / 17

Detecting Errors in Introduction & Motivation Semantic Annotation Corpora with semantic annotation are increasingly relevant Introduction & Motivation in natural language processing Background ◮ See: Baker et al. (1998); Palmer et al. (2005); Burchardt Detecting semantic annotation errors et al. (2006); Taul´ e et al. (2005) Argument labeling variation Argument identification variation Heuristics for disambiguating strings Evaluation Results Insights Summary & Outlook References 2 / 17

Detecting Errors in Introduction & Motivation Semantic Annotation Corpora with semantic annotation are increasingly relevant Introduction & Motivation in natural language processing Background ◮ See: Baker et al. (1998); Palmer et al. (2005); Burchardt Detecting semantic annotation errors et al. (2006); Taul´ e et al. (2005) Argument labeling variation Argument identification Semantic role labeling variation Heuristics for disambiguating strings ◮ used for tasks such as: Evaluation ◮ information extraction (Surdeanu et al. 2003) Results ◮ machine translation (Komachi et al. 2006) Insights Summary & Outlook ◮ question answering (Narayanan and Harabagiu 2004) References 2 / 17

Detecting Errors in Introduction & Motivation Semantic Annotation Corpora with semantic annotation are increasingly relevant Introduction & Motivation in natural language processing Background ◮ See: Baker et al. (1998); Palmer et al. (2005); Burchardt Detecting semantic annotation errors et al. (2006); Taul´ e et al. (2005) Argument labeling variation Argument identification Semantic role labeling variation Heuristics for disambiguating strings ◮ used for tasks such as: Evaluation ◮ information extraction (Surdeanu et al. 2003) Results ◮ machine translation (Komachi et al. 2006) Insights Summary & Outlook ◮ question answering (Narayanan and Harabagiu 2004) References ◮ requires corpora annotated with predicate-argument structure for training and testing data ◮ Gildea and Jurafsky (2002); Xue and Palmer (2004); Toutanova et al. (2005); Pradhan et al. (2005), ... 2 / 17

Detecting Errors in Introduction & Motivation Semantic Annotation Corpora with semantic annotation are increasingly relevant Introduction & Motivation in natural language processing Background ◮ See: Baker et al. (1998); Palmer et al. (2005); Burchardt Detecting semantic annotation errors et al. (2006); Taul´ e et al. (2005) Argument labeling variation Argument identification Semantic role labeling variation Heuristics for disambiguating strings ◮ used for tasks such as: Evaluation ◮ information extraction (Surdeanu et al. 2003) Results ◮ machine translation (Komachi et al. 2006) Insights Summary & Outlook ◮ question answering (Narayanan and Harabagiu 2004) References ◮ requires corpora annotated with predicate-argument structure for training and testing data ◮ Gildea and Jurafsky (2002); Xue and Palmer (2004); Toutanova et al. (2005); Pradhan et al. (2005), ... Semantically-annotated corpora also have potential as sources of linguistic data for theoretical research 2 / 17

Detecting Errors in Exploring semantic annotation Semantic Annotation Need feedback on annotation schemes: Introduction & Motivation ◮ difficult to select an underlying theory (see, e.g., Background Burchardt et al. 2006) Detecting semantic annotation errors ◮ difficult to determine certain relations, e.g., modifiers Argument labeling variation Argument identification (ArgM) in PropBank (Palmer et al. 2005) variation Heuristics for disambiguating strings Evaluation Results Insights Summary & Outlook References 3 / 17

Detecting Errors in Exploring semantic annotation Semantic Annotation Need feedback on annotation schemes: Introduction & Motivation ◮ difficult to select an underlying theory (see, e.g., Background Burchardt et al. 2006) Detecting semantic annotation errors ◮ difficult to determine certain relations, e.g., modifiers Argument labeling variation Argument identification (ArgM) in PropBank (Palmer et al. 2005) variation Heuristics for disambiguating strings Need to detect annotation errors, which can: Evaluation ◮ harmfully affect training (e.g., van Halteren et al. 2001; Results Insights Dickinson and Meurers 2005b) Summary & Outlook ◮ harmfully affect evaluation (Padro and Marquez 1998; References Kvˇ etˇ on and Oliva 2002) 3 / 17

Detecting Errors in Exploring semantic annotation Semantic Annotation Need feedback on annotation schemes: Introduction & Motivation ◮ difficult to select an underlying theory (see, e.g., Background Burchardt et al. 2006) Detecting semantic annotation errors ◮ difficult to determine certain relations, e.g., modifiers Argument labeling variation Argument identification (ArgM) in PropBank (Palmer et al. 2005) variation Heuristics for disambiguating strings Need to detect annotation errors, which can: Evaluation ◮ harmfully affect training (e.g., van Halteren et al. 2001; Results Insights Dickinson and Meurers 2005b) Summary & Outlook ◮ harmfully affect evaluation (Padro and Marquez 1998; References Kvˇ etˇ on and Oliva 2002) Little work on automatically detecting errors in semantically-annotated corpora ◮ Mainly POS and syntactically-annotated corpora (see Dickinson 2005, ch. 1) 3 / 17

Detecting Errors in Background: the variation n-gram method Semantic Annotation Dickinson and Meurers (2003a) Introduction & Variation : material occurs multiple times in corpus Motivation Background with different annotations Detecting semantic annotation errors Argument labeling variation Argument identification variation Heuristics for disambiguating strings Evaluation Results Insights Summary & Outlook References 4 / 17

Detecting Errors in Background: the variation n-gram method Semantic Annotation Dickinson and Meurers (2003a) Introduction & Variation : material occurs multiple times in corpus Motivation Background with different annotations Detecting semantic annotation errors Dickinson and Meurers (2003a) introduces the notions Argument labeling variation Argument identification variation ◮ variation nucleus : recurring word with different annotation Heuristics for disambiguating strings ◮ variation n-gram : variation nucleus with identical context Evaluation Results and provides an efficient algorithm to compute them. Insights Summary & Outlook References 4 / 17

Detecting Errors in Background: the variation n-gram method Semantic Annotation Dickinson and Meurers (2003a) Introduction & Variation : material occurs multiple times in corpus Motivation Background with different annotations Detecting semantic annotation errors Dickinson and Meurers (2003a) introduces the notions Argument labeling variation Argument identification variation ◮ variation nucleus : recurring word with different annotation Heuristics for disambiguating strings ◮ variation n-gram : variation nucleus with identical context Evaluation Results and provides an efficient algorithm to compute them. Insights Summary & Outlook Example: 12-gram with variation nucleus off References (1) to ward off a hostile takeover attempt by two European shipping concerns In the two occurrences of this 12-gram in the WSJ, off is ◮ once annotated as a preposition (IN), and ◮ once as a particle (RP). 4 / 17

Detecting Errors in Heuristics for disambigutation Semantic Annotation Introduction & Motivation Variation can result from: Background Detecting semantic ◮ ambiguity : different possible labels occur in different annotation errors Argument labeling variation corpus occurrences Argument identification variation ◮ error : labeling of a string is inconsistent across Heuristics for disambiguating strings comparable occurrences Evaluation Results Insights Summary & Outlook References 5 / 17

Detecting Errors in Heuristics for disambigutation Semantic Annotation Introduction & Motivation Variation can result from: Background Detecting semantic ◮ ambiguity : different possible labels occur in different annotation errors Argument labeling variation corpus occurrences Argument identification variation ◮ error : labeling of a string is inconsistent across Heuristics for disambiguating strings comparable occurrences Evaluation Results Insights Non-fringe heuristic to detect annotation errors: Summary & Outlook ◮ Nuclei found at fringe of n -gram more likely to be References genuine ambiguities (Dickinson 2005) 5 / 17