Social Media Argumentation Mining: The Quest for Deliberateness in - PowerPoint PPT Presentation

Social Media Argumentation Mining: The Quest for Deliberateness in Raucousnes Jan Šnajder Joint work with Filip Boltuži´ c Text Analysis and Knowledge Engineering Lab FER, University of Zagreb Dagstuhl 19 April 2016 1 / 36

Social media argumentation mining Much work on AM focuses on well-structured, edited text: e.g. legal documents [Walton, 2005] and scientific documents [Jiménez-Aleixandre and Erduran, 2007] Recent interest in AM from social media texts : online debates [Cabrio and Villata, 2012, Habernal et al., , Boltuži´ c and Šnajder, 2014], discussions on regulations [Park and Cardie, 2014], and product reviews [Ghosh et al., 2014] Online debates – a somewhat controlled setting Comment boards, product reviews, microblogs – less controlled 2 / 36

Outline Argumentation in social media 1 Argument recognition 2 Argument clustering 3 Argument similarity 4 3 / 36

Outline Argumentation in social media 1 Argument recognition 2 Argument clustering 3 Argument similarity 4 4 / 36

User comments Our interest: AM from user comments (not necessarily debates) Yahoo News: User comment on Trump rally event The President we have now divided our country and put his ego first instead of the people. Trump hasn’t divided the country that’s why he has so many people behind him. We want someone who is not afraid of the politics in Washington and change our policies with dealing with other countries. No predefined topic (topics emerge ad hoc) Mostly monological 5 / 36

Why analyze this? To the extent in which we are interested in analyzing opinion of other people, we should also be interested in analyzing the underlying reasons to fully apprehend their opinions Our (long term) goal Analyze, on a large scale, what arguments people use to express their stance, including the faulty arguments, as well as all the (often implicit) premises (reflecting beliefs, policies, value systems, . . . ) these arguments build on. 6 / 36

Challenges of social media AM Noisy text Vague claims (unclear, ambiguous, poorly worded) Vague/incomplete argument structure (esp. true of short texts) 7 / 36

Main tasks? Component identification – the task of detecting the premises and conclusion of an argument, as found in a text of discourse Relation prediction – identifying the relations between components [Habernal and Gurevych, 2016]: a (slightly modified) Toulmin model may be suitable for short documents, such as article comments or forum posts Relevant tasks, but it’s not obvious how they help in analyzing user arguments on a large scale, where we need to be able to determine the identity of arguments (expressed in text) 8 / 36

Main tasks for social media AM? Identify the main arguments – identify the main (central, most prominent, most often used) arguments that the users use when discussing a certain topic Classify opinionated posts – given an opinionated user comment, identify the main arguments used in it 9 / 36

Example Yahoo News: User comment on Trump rally event The President we have now divided our country and put his ego first instead of the people. Trump hasn’t divided the country that’s why he has so many people behind him. We want someone who is not afraid of the politics in Washington and change our policies with dealing with other countries. Main argument: “Donald Trump would make a good president” Main claim: “Donald Trump will change the foreign policy for the better” Premises: “Existing foreign policy is bad” , “Trump is not afraid to take on the Establishment” , etc. 10 / 36

Machine learning perspective Argument clustering – grouping of similar arguments, so that the main arguments/claims can be identified [Boltuži´ c and Šnajder, 2015] Argument classification – given an opinionated comment, classify it into one or many classes, each corresponding to one main argument (obtained either manually or using argument clustering) [Hasan and Ng, 2014]: “reason classification”, [Boltuži´ c and Šnajder, 2014]: “argument recognition” 11 / 36

Outline Argumentation in social media 1 Argument recognition 2 Argument clustering 3 Argument similarity 4 12 / 36

Task Description Argument Recognition Identifying what arguments, from a set of predefined arguments, are used in a comment, and how. Input: 1 Prominent arguments from past two-sided debates 2 Users’ comments from on-line discussion boards Output (for each comment): 1 Is an argument used in a comment? 2 Does the comment support or attack the argument? 13 / 36

Should gay marriage be legal? Comment Gay marriages must be legal in all 50 states. 2 people regardless of their genders. Discrimination against gay marriage is unconstitutional and biased. Tolerance, education and social justice make our world a better place. � Supported argument It is discriminatory to refuse gay couples the right to marry � Attacked argument Marriage should be between a man and a woman. 14 / 36

C OM A RG Corpus C OM A RG : Corpus of comments, arguments, and manually annotated comment–argument pairs Source: procon.org and idebate.org Under God in Pledge (UGIP) Gay Marriages (GM) # Arguments 6 7 # Comments 175 198 # Pairs 1,050 1,386 Five-point scale: � A – comment explicitly attacks the argument � a – comment vaguely/implicitly attacks the argument � N – comment makes no use of the argument � s – comment vaguely/implicitly supports the argument � S – comment explicitly supports the argument 15 / 36

Annotation Example Comment I believe that the statement about God in the pledge should be eliminated. In order to create unity in our nation we shouldn’t be forcing someone else’s God onto people. Also, adding the phrase Under God" was a decision made to widen the gap between us and the Soviet Union. It wasn’t put there to "honor god" or make us any better. Furthermore, we should seperate church from state. Its the law. � S (explicitly supported) Separation of state and religion. � a (vaguely/implicitly attacked) Under God is part of American tradition and history. � N (not used) Likely to be seen as a state sanctioned condemnation of religion. 16 / 36

Argument Recognition Model Argument Recognition framed as multiclass classification Features: 1 Textual Entailment (TE) Excitement Open Platform: 7 pre-trained decision algorithms (14 features) 2 Semantic Text Similarity (STS) TakeLab STS sentence-level, comment-level (32 features) 3 Stance Alignment (SA) Binary feature: 1 if argument and comment have same stance No lexical features ⇒ topic independence 17 / 36

Results Micro-averaged F1-score A-a-N-s-S Aa-N-sS A-N-S Model UGIP GM UGIP GM UGIP GM MCC baseline 68.2 69.4 68.2 69.4 79.5 76.6 BoWO baseline 68.2 69.4 67.8 69.5 79.6 76.9 TE 69.1 69.6 72.3 80.1 73.4 81.1 STS 67.8 68.7 67.3 69.9 79.2 75.8 SA 68.2 69.4 68.2 69.4 79.5 76.6 STS+SA 68.2 69.5 67.5 68.7 79.6 76.1 TE+SA 68.9 72.4 71.0 73.7 81.8 80.3 TE+STS+SA 72.5 68.9 73.4 81.4 79.7 70.5 STS or STS+SA not good TE outperforms baseline from 0.6% to 11.7% F1 TE+SA overall best SA helps distinguish entailment/contradiction 18 / 36

Outline Argumentation in social media 1 Argument recognition 2 Argument clustering 3 Argument similarity 4 19 / 36

Should marijuana be legalized? User comment 1 No, because marijuana lessen the brain’s ability for cognitive thinking. User comment 2 There have been plenty of highway deaths associated with marajuanna use. User comment 3 The Legalization of marijuana would lower are crime rates in the United States of America by at least 15 to 20 User comment 4 Marijuana is proven to cause depression and change brain patterns in odd ways among other things 20 / 36

Should marijuana be legalized? “Damages health" (CON) User comment 1 No, because marijuana lessen the brain’s ability for cognitive thinking. User comment 4 Marijuana is proven to cause depression and change brain patterns in odd ways among other things 21 / 36

Task Description Identifying Prominent Arguments Identifying reasonings and opinions to cluster into arguments. Input: 1 Users’ comments from on-line discussions Output: 1 Set of argument clusters 2 Representative argument of each cluster 22 / 36

Corpus Threaded debated annotated with arguments at sentence level [Hasan and Ng, 2014] Four topics Should gay marriage be legal? Should marijuana be legalized? Is Obama a good president? Should abortion be legalized? GM MAR OBA ABO Pro Con Pro Con Pro Con Pro Con #Arguments 5 4 5 5 8 8 7 5 #Comments 639 197 585 239 358 272 446 368 23 / 36

Argument similarity 1 Vector-space similarity Bag-of-words (BoW) Inverse sentence frequency weight Neural network skip-gram [Mikolov et al., 2013] Word-vector sum for sentences Cosine distance 2 Semantic textual similarity (STS) [Šari´ c et al., 2012] Text comparison features Output real valued similarity score Hierarhical agglomerative clustering (HAC) [Xu et al., 2005] Input : Distance matrix Output : Hierarhical structures Linkage criterion: Complete linkage, Ward’s method 24 / 36