Automated Argumentation Analysis Gijsbert Erkens Department of Education Utrecht University April 15 Naples, webinar
2. Setup of the webinar 1. Communicative markers in collaboration dialogues: 1. Discourse markers 2. Content markers 2. Automatic coding of communicative markers 3. Dialogue acts & argumentation 4. Reliability & validity in automatic coding 5. Use of automatic coding: 1. Analysis 2. Assessment 3. (online) Feedback 2
3. Communicative markers in dialogues Discussion: • How would you analyze the collaboration dialogue fragment on the next slide? • What is happening? • What is relevant for collaboration? • How is the process of collaboration? • Style of collaboration? • Etc. 3
4. Dialogue fragment (chat) • 3 girls, history essay writing : “Strong women in Antiquity” (sources about Andromache, Electra, etc., & interviews with nowadays ‘strong’ women). They are working on sub question 1b: What did the ‘strong’ woman do? . • 504: I didn’t really find deeds, I have told about her dilemma • 505: Oh, well… • 506: No, I didn’t either. But, would you guys please check what I have written? Because, maybe I’m fully wrong or whatever,.. • 505: No, girl, of course not • 504: It is probably okay • 505: I’ll read it now • 506: Ok, thnx • 505: You did fine • 504: It is good, I think. But, sorry to say so, first you write Hector with a c and later with a k! • 506: Oops! I’ll change the k! • 506: But I think her deeds and her role in society are almost the same as far as I can describe it. • 505: Yes, well we say: see also sub question 1d or something like that. • 506: Yeah, okay 4
5. Coordination tasks in collaboration On different layers: • Task content (approach, strategy, knowledge) • Collaboration (task division, negotiation, integration) • Communication (understanding, grounding) • Social-relation (sharing, communality) • Motivation (persistence, interest) • Emotion (frustration, coping) Students coordinate and solve coordination problems on these different layers in a direct or a meta-cognitive way. The coordination platform is the dialogue and multifunctionality of speech makes it possible. 5
6. Discourse and content markers • Task content content concepts uncertainty (sort of, I think, etc.) • Collaboration task division (we should, you do, etc.) argumentation (dialogue acts) • Communication understanding (what? do you mean..) • Social-relation personal (I, me, you, your, we, our) • Motivation interest (boring, stupid, difficult) • Emotion frustration (smileys, exclamation marks) 6
7. Automatic coding of communicative markers • Two approaches on automatic coding: • Knowledge free (data mining, pattern mining) • Knowledge based (finding predefined patterns) Discussion: • Advantages & disadvantages of automatic coding? • Advantages & disadvantages of both approaches ? 7
8. Analysis of collaboration dialogue protocols • Qualitative, interpretative analysis of relevant phenomena & categories in the protocols • Necessary, but • Problem of subjectivity • Problem of biased searching, • Development of systematic coding system • Problem of unit of coding, segmentation problem • Exhaustive, exclusive and independent coding • Reliability (interrater agreement, Cohen’s kappa) • Validity problems (seldom addressed) • Coding of protocols • Problem of robustness • Problem of reliability (stability) • Problem of tediousness, labor intensiveness 8
9. Automatic coding of communicative function Assumptions: • Every utterance has a communicative function, fulfills a pragmatic action (Taylor, 1990) • i.e. Searle (1969) Speech acts: Assertives, Directives, Commissives, Expressives, Declarations • The communicative, pragmatic function of utterances are being signaled by language users by explicit ‘discourse markers’ (Schiffrin, 1987) • ‘Oh’, ‘By the way,’, ‘Well’, ‘However,’, So,’ • Discourse markers are used to support the coherence in discourse: they signal how the utterance should be interpreted in the context of the ongoing discourse. 9
10. MEPA, Multiple Episode Protocol Analysis 10
11. MEPA • Protocol scripting, annotation & coding • Dialogues, discussions or (inter)actions • dynamic verbal or nonverbal data • Qualitative & statistical online analysis • Frequency, cross table, interrater, lag sequential, sorting, visual chart, word concordance, etc. • Multidimensional / hierarchical • Flexible, explorative environment • (Semi)-automatic coding • Free to use: G.Erkens@uu.nl 11
Dialogue acts and argumentation • Communicative, pragmatic functions of dialogue utterances • Argumentatives (convincing the other) • Informatives (information transfer to other) • Responsives (reacting to the other) • Elicitatives (eliciting reaction from other) • Imperatives (commanding the other) Discussion: • Is every argumentative discourse marker (e.g. but, because) really meant to convince the other? 12
13. Dialogue Act Coding Communicative Dialogue act Specification Code Description Discourse marker, function i.e. Argumentatives Reason ArgRsn Reason, ground “Because …” Contra ArgCnt Counterargument “However, …” Reasoning Conditional ArgCon Condition “If …” Then ArgThn Consequence “Then …” Disjunctive ArgDis Disjunctive “Or …” Conclusion ArgCcl Conclusion “So, …” Elaboration ArgEla Continuation “Furthermore, …” Responsives Confirmation ResCfm Confirmation of info “Right” Deny ResDen Refutation of info “No” Reaction, or Acceptation ResAcc Acceptance of info “Oh” response to an Reply Confirm ResRplCfm Affirmative reply “Sure” utterance Deny ResRplDen Negative reply “No way” to an Accept ResRplAcc Accepting reply “Okay” elicitative Statement ResRplStm Statement reply “ …” Performative ResRplPer Performative reply “Thanks” 13
14. Dialogue Act Coding Communicative Dialogue act Specification Code Description Discourse marker, function i.e. Informatives Performative Action performed by “Hello” InfPer saying it Transfer of Evaluation Neutral InfEvlNeu Neutral evaluation “…easy …” information Positive InfEvlPos Positive evaluation “Nice!” Negative InfEvlNeg Negative evaluation “Awful …” Statement InfStm Task information “ …” Action InfStmAct Announcement of actions “I’ll do …” Social InfStmSoc Social statement “Love you …” Nonsense InfStmNon Nonsense statement “grrumppphh” Question Verify EliQstVer Yes/no question “Agree?” Elicitatives Set Set question/ multiple “…. or….?” EliQstSet Utterances choice requiring a Open EliQstOpn Open question “Why?” response Proposal Action EliPrpAct Proposal for action “Let’s change …” Action ImpAct Order for action “W8!” Imperatives Commanding Focus ImpFoc Order for attention “Watch!” utterances 14
15. Automatic coding of dialogue acts in chats • Segmentation filter (SEG filter): 300 production rules • punctuation characters (i.e. ‘?’,’!’, ‘.’) • connectives (‘however,’, ‘so,’, ’but’) • starting discourse markers (‘well’, ‘on the other hand’) • Exception or restriction rules • Splitting in messages before or after marker • Dialogue Act Coding (DAC filter): 1250 production rules • Coding messages on discourse markers of communicative, pragmatic function • InfStm? as default catch-all • 29 Dialogue Acts 15
16. Reliability and validity of automatic coding • Reliability • Is the automatic coding procedure reliable? • Validity Rourke and Anderson (2004) validity of coding by: examination of group differences, examination of experimental intervention and correlational analyses: 1. Can the automatic coding procedure be validated through examination of group differences? 2. Can the automatic coding procedure be validated through examination of experimental intervention? 3. Can the automatic coding procedure be validated through correlation analyses? Discussion: • The same reliability and validity questions to manual interpretative or systematic coding systems? 16
17. VCRI groupware environment 17
18. Reliability: Is the automatic coding procedure reliable? • DAC filter will apply the same rules in the same way every time over • Error analysis: • Interrater reliability analysis comparing hand coding and automatic coding on dialogue acts of the same protocol. • Over 500 messages: • Interrater agreement percentage (human–computer): 96 % • Cohen’s kappa: .78 18
19. Validity 1 : gender differences • Female students utter significantly more dialogue acts than male students • Multilevel analysis (correcting for number of messages): • Female students: more argumentatives, especially reasons (ArgRsn) & conclusions (ArgCcl) • Female students: more responsives, especially confirmations (ResCfm & ResRplCfm) • Male students: more informatives, especially statements (InfStm) and nonsense (InfStmNon) • Male students: more negative evaluations (InfEvlNeg) • Male students: more imperatives, especially focusing attention imperatives (ImpFoc) • Expectations partly confirmed (female students use more arguments) 19
Recommend
More recommend
