Argumentative Writing Support: Structure Identification and Quality - PowerPoint PPT Presentation

Argumentative Writing Support: Structure Identification and Quality Assessment of Arguments Dagstuhl Seminar: “Natural Language Argumentation: Mining, Processing, and Reasoning over Textual Arguments” Christian Stab Iryna Gurevych Ubiquitous Knowledge Processing Lab Technische Universität Darmstadt 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 1

Argumentation Mining Research @ Darmstadt  Analyzing argumentation structures in the discourse context , e.g. user- generated Web content (Habernal & Gurevych, 2015), scientific articles, (Kirschner et al., 2015), student essays (Stab & Gurevych, 2014)  Projects:  Argumentative Writing Support (AWS): a writing assistance system to support authors in writing persuasive arguments and to improve writing skills This talk: looking at persuasive essays (PhD thesis by Christian Stab)  Large-scale argumentation mining on the Web: e.g. comments to articles, discussion forums, or blogs (joint project with Ivan, Benno, Henning)  Information Consolidation on the Web: correspondence between proposition- level semantic relations and argumentative structures (joint project with Ido)  Free resources, e.g. argument-annotated corpora, guidelines, software : https://www.ukp.tu-darmstadt.de/research/research- areas/argumentation-mining/ 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 2

Outline Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 5

Parsing Argumentation Structures Example Example Essay: Argumentation Structure: Museums and art galleries will disappear soon? Major Claim(s) 1 &13 It is quite common that more and more people can watch exhibitions through television or internet at home due to modern technology; therefore, some people think museums and art galleries will disappear soon. However, I still believe [some museums and art galleries will not disappear] 1 . [Technology indeed simplifies people's life all the time] 2 . Claims 3 6 11 12 Obviously, [people who watch exhibitions on TV or internet at home, save the time and money on the road, which is increasingly significant particularly to people in modern society] 3 . However, [in accordance with recent research, experts suggest the lifestyle of individuals in modern society is unhealthy] 4 because [they lack of physical exercise and face- to-face communication] 5 . Premises 2 4 7 9 10 [The importance of museums and art galleries is plain in terms of education and culture] 6 . First of all, [authentic exhibits cannot be completely displayed only by images and videos] 7 . [Travelling to a place is much better than viewing the landscape 5 8 of that place on TV or photos] 8 , so [the best method to learn one thing is to experience it] 9 . Furthermore, [museums and art Paragraph Paragraph Paragraph galleries preserve some culture heritages] 10 ; therefore, [these 2 3 4 buildings will not disappear unless people abandon their culture] 11 . In conclusion, I admit that [modern technology has provided a more convenient and comfortable manner for people to watch exhibitions] 12 but [museums and art galleries are necessary to be preserved for its importance of education and culture] 13 . 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 6

Parsing Argumentation Structures Data Persuasive essays  Written for e.g. IELTS, TOEFL, etc.  Collected from www.essayforum.com  402 essays; 7,116 sentences; 147,271 tokens Annotation scheme  Argumentative structure as tree structure  Argument components: Major Claim (751) , Claim (1,506) and Premise (3,832)  Argumentative relations: Support (3,613) and Attack (219) Inter-Annotator Agreement*  Argument components: α U =.767  Argumentative relations: α = .723 (avg. of support & attack) *determined among three annotators on a subset of 80 essays 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 7

Parsing Argumentation Structures Pipeline Argument Component Classification  Classify each argument component as major claim, claim or premise Argumentative Relation Identification  Classify argument component pairs as argumentatively related or not Problem: Result is an arbitrary graph NOT a tree Solution: Joint Modeling (Tree generation) 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 8

Parsing Argumentation Structures Joint Modeling Component types and argumentative relations share mutual information Component Type Argumentative Relation Claim No outgoing relations (root node) Premise Exhibits outgoing relations Claim More incoming relations Premise Fewer incoming relations Idea : Jointly model argument component types and argumentative relations to find an optimal tree ILP-based joint model  Finds the tree structure which optimizes previous analysis results  Allows to find several trees (arguments) in a paragraph 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 9

Parsing Argumentation Structures Joint Modeling component classification relation identification statistics Cl → F1 MC Cl Pr F1 NoLi Link Pr → Cl Trees Pr Baseline heuristics .724 .740 .560 .870 .660 .885 .436 - - 100 % † † Base classifier .773 . 865 .592 .861 .736 . 917 .547 - - 20.9 % † † IncBaseline .776 . 865 .601 .861 .739 . 917 .555 206 1,144 24.2 % †‡ † ILP Joint Model . 823 . 865 . 701 . 904 . 752 .913 . 591 297 283 100 % Baseline Heuristic  Last component in introduction and first component in conclusion as major claim  First component in paragraph as claim, remaining as premise  Link all premises to first component in paragraph Base classifier  Argument Component Classification (major claim, claim, premise)  Classification of argument component pairs (linked, unlinked) IncBaseline  Incorporates baseline in base classifiers if both base classifiers fail to predict claims or relations in a paragraph † significant improvement over baseline heuristic; ‡ significant improvement over base classifier 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | 10

Outline Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 11

Myside Bias Recognition Introduction & Motivation Myside Bias  Tendency to ignore evidence against one’s own position  Argumentation is biased towards own prior beliefs  Weak arguments Considering opposing viewpoints is crucial  Improves precision of claims  Better elaboration of reasons  Significantly improves the argumentation quality (Wolfe and Britt, 2009) Myside Bias Recognition  Detecting text/arguments that anticipate opposing viewpoints  Applications: filtering bad arguments, writing support, etc. 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 12

Myside Bias Recognition Data & Approach Data Task  402 persuasive essays  Binary Document Classification Inter-Annotator Agreement Learner  κ = .786  SVM with polykernel  α = .787 Features Class Distribution  Unigrams  37.6% unbiased  Dependencies  62.4% biased  Production Rules  myside bias is a frequent flaw  Adversative transitions  Sentiment features  Discourse features 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 13

Myside Bias Recognition Results – Feature Analysis on Development Set 0,8 0,75 0,709 0,657 0,6 0,569 Macro F1 0,52 0,4 0,385 0,2 0 Dependencies Unigram Depencies Producation Rules Adverstative Sentiment Features Discourse Features Transitions  Adversative Transitions and unigrams perform best  Sentiment features are not informative; same results as majority baseline  Myside bias is indicated by lexical features 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 14

Myside Bias Recognition Results – Model Assessment 1 0,894 0,8 0,734 0,679 Macro F1 0,6 0,4 0,384 0,2 0 Human Upper Bound Baseline Majority Baseline Heuristics SVM uni+pr+adv  Baseline Majority : Classifies each text as biased  Baseline Heuristics : All texts with ‘ Admittedly ’ or ‘ argue that ’ are unbiased  SVM uni+pr+adv : SVM with unigrams, production rules and adversative transitions  Yields best performance (75.6% accuracy; 0.734 macro F1)  Significantly outperforms heuristic baseline (Wilcoxon sign ranked test; sign level = 0.005)  Achieves 84% of human performance (accuracy) 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 15

Outline Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab | 16