an interpretable joint graphical model for fact checking
play

An Interpretable Joint Graphical Model for Fact-Checking from Crowds - PowerPoint PPT Presentation

Aug 17, 2023 •354 likes •780 views

An Interpretable Joint Graphical Model for Fact-Checking from Crowds An T. Nguyen 1 Aditya Kharosekar 1 Matthew Lease 1 Byron C. Wallace 2 1 University of Texas at Austin 2 Northeastern University 1 Problems Given a claim: Facebook Shut Down an

  1. An Interpretable Joint Graphical Model for Fact-Checking from Crowds An T. Nguyen 1 Aditya Kharosekar 1 Matthew Lease 1 Byron C. Wallace 2 1 University of Texas at Austin 2 Northeastern University 1

  2. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. 2

  3. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com 2

  4. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com Predict headline stance: For Against Observing 2

  5. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com Predict headline stance: For Against Observing Predict claim veracity: False True Unknown 2

  6. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com Predict headline stance: For Against Observing Predict claim veracity: False True Unknown 2

  7. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com Predict headline stance: For Against Observing Predict claim veracity: False True Unknown Our motivation: ◮ Make sense of general claims incl. scientific, historical, ... 2

  8. Problems Given a claim: Facebook Shut Down an AI Experiment Because Chatbots Developed Their Own Language. and relevant article headlines: No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. source: gizmodo.com Predict headline stance: For Against Observing Predict claim veracity: False True Unknown Our motivation: ◮ Make sense of general claims incl. scientific, historical, ... ◮ Not just “fake news”. 2

  9. Solutions Previous work: 3

  10. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). 3

  11. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). ◮ Predict veracity from stance+source features (Popat et al. 2017) 3

  12. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). ◮ Predict veracity from stance+source features (Popat et al. 2017) We proposed: ◮ Crowdsource stance labels. 3

  13. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). ◮ Predict veracity from stance+source features (Popat et al. 2017) We proposed: ◮ Crowdsource stance labels. ◮ Hybrid human AI ◮ Available near real-time 3

  14. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). ◮ Predict veracity from stance+source features (Popat et al. 2017) We proposed: ◮ Crowdsource stance labels. ◮ Hybrid human AI ◮ Available near real-time ◮ Joint graphical model of stance, veracity, annotators. 3

  15. Solutions Previous work: ◮ Predict stance from text features (Ferreira& Vlachos 2016). ◮ Predict veracity from stance+source features (Popat et al. 2017) We proposed: ◮ Crowdsource stance labels. ◮ Hybrid human AI ◮ Available near real-time ◮ Joint graphical model of stance, veracity, annotators. ◮ Interaction between variables ◮ Interpretable 3

  16. Powered by TCPDF (www.tcpdf.org) Powered by TCPDF (www.tcpdf.org) Model B U L A c lablers V S W T n claims R m sources 4

  17. Powered by TCPDF (www.tcpdf.org) Powered by TCPDF (www.tcpdf.org) Model B U L A c lablers V S W 1. Predict Stance S ◮ Text features T T n claims R m sources Powered by TCPDF (www.tcpdf.org) 4

  18. Powered by TCPDF (www.tcpdf.org) Powered by TCPDF (www.tcpdf.org) Model B U L A c lablers V S W 1. Predict Stance S ◮ Text features T T n claims 2. Predict Veracity V R ◮ Stance S m sources Powered by TCPDF (www.tcpdf.org) ◮ Reputation R 4

  19. Model B 1. Predict Stance S ◮ Text features T U L A 2. Predict Veracity V c lablers ◮ Stance S V S W ◮ Reputation R 3. Stance Label L T ◮ True stance S n claims ◮ Annotator competence A R m sources Powered by TCPDF (www.tcpdf.org) Powered by TCPDF (www.tcpdf.org) 4

  20. Inference & Learning Inference: ◮ Gibbs sampling: accurate but slow. 5

  21. Inference & Learning Inference: ◮ Gibbs sampling: accurate but slow. ◮ Variational inference: fast but biased. 5

  22. Inference & Learning Inference: ◮ Gibbs sampling: accurate but slow. ◮ Variational inference: fast but biased. Learning: Expectation Maximization. 5

  23. Inference & Learning Inference: ◮ Gibbs sampling: accurate but slow. ◮ Variational inference: fast but biased. Learning: Expectation Maximization. Details in the paper. 5

  24. Evaluation Data: Emergent (Ferreira and Vlachos 2016) ◮ 300 claims. ◮ 2595 articles with stance labels. 6

  25. Evaluation Data: Emergent (Ferreira and Vlachos 2016) ◮ 300 claims. ◮ 2595 articles with stance labels. ◮ We collected: crowd stance labels by Mechanical Turk. 6

  26. Evaluation Data: Emergent (Ferreira and Vlachos 2016) ◮ 300 claims. ◮ 2595 articles with stance labels. ◮ We collected: crowd stance labels by Mechanical Turk. Baseline: Separated models for stance, veracity & crowd labels. 6

  27. Evaluation Data: Emergent (Ferreira and Vlachos 2016) ◮ 300 claims. ◮ 2595 articles with stance labels. ◮ We collected: crowd stance labels by Mechanical Turk. Baseline: Separated models for stance, veracity & crowd labels. Metric: Brier score, measures accuracy and prob. calibration. 6

  28. Results 7

  29. User study Interface: users enter claims, see predictions. 8

  30. User study Interface: users enter claims, see predictions. A/B testing 8

  31. User study Interface: users enter claims, see predictions. A/B testing ◮ A: see only veracity predictions 8

  32. User study Interface: users enter claims, see predictions. A/B testing ◮ A: see only veracity predictions ◮ B: also see explanation (reputation, stances) 8

  33. User study Interface: users enter claims, see predictions. A/B testing ◮ A: see only veracity predictions ◮ B: also see explanation (reputation, stances) 8

  34. User study: results 9

  35. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. 10

  36. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. 10

  37. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. Paper: experiments on Snopes. 10

  38. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. Paper: experiments on Snopes. Demo: fcweb.pythonanywhere.com 10

  39. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. Paper: experiments on Snopes. Demo: fcweb.pythonanywhere.com We share code + data 10

  40. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. Paper: experiments on Snopes. Demo: fcweb.pythonanywhere.com We share code + data Acknowledge: Crowd annotator, reviewers, NSF. 10

  41. Conclusion Takeaway: ◮ Stance/Veracity predictions are hard. ◮ We contribute: crowdsourcing + joint modeling. Paper: experiments on Snopes. Demo: fcweb.pythonanywhere.com We share code + data Acknowledge: Crowd annotator, reviewers, NSF. Questions? 10

Recommend

Statistical Statistical Statistical Model Statistical Model Model Checking Model Checking

Statistical Statistical Statistical Model Statistical Model Model Checking Model Checking

Statistical Statistical Statistical Model Statistical Model Model Checking Model Checking Checking Checking for Timed Automata Timed Automata Coll C l C ll b llabora orators: t ors: Peter Bulychev, Alexandre David Axel Legay, Marius

725 views • 59 slides
Hoare Logic and Model Checking Model Checking Lecture 11: Model checking for Computation Tree

Hoare Logic and Model Checking Model Checking Lecture 11: Model checking for Computation Tree

Hoare Logic and Model Checking Model Checking Lecture 11: Model checking for Computation Tree Logic Dominic Mulligan Based on previous slides by Alan Mycroft and Mike Gordon Programming, Logic, and Semantics Group University of Cambridge

402 views • 25 slides
Handling Loops in Bounded Model Checking of C Programs via k- Induction Lucas Cordeiro Joint

Handling Loops in Bounded Model Checking of C Programs via k- Induction Lucas Cordeiro Joint

Handling Loops in Bounded Model Checking of C Programs via k- Induction Lucas Cordeiro Joint work with Jeremy Morse, Mikhail Ramalho, Herberto Rocha, Hussama Ismail, Raimundo Barreto, Denis Nicole, and Bernd Fischer Bounded Model Checking

553 views • 37 slides
Dynamic Sampling fs om Graphical Models Yitong Yin Nanjing University Joint work with W

Dynamic Sampling fs om Graphical Models Yitong Yin Nanjing University Joint work with W

Dynamic Sampling fs om Graphical Models Yitong Yin Nanjing University Joint work with W eiming Feng ( Nanjing ) Nisheeth Vishnoi ( EPFL ) Graphical Model instance of graphical model: I = ( V, E, [ q ] , ) V : variables v E 2

936 views • 55 slides
SMT-based Software Model Checking: Experimental Comparison of Four Algorithms Matthias Dangl

SMT-based Software Model Checking: Experimental Comparison of Four Algorithms Matthias Dangl

SMT-based Software Model Checking: Experimental Comparison of Four Algorithms Matthias Dangl Joint work with Dirk Beyer University of Passau, Germany SMT-based Software Model Checking Bounded Model Checking ( Cbmc , CPAchecker , Esbmc ,

496 views • 26 slides
Recent Advances in Automated Fact Checking Immanuel Trummer Cornell University

Recent Advances in Automated Fact Checking Immanuel Trummer Cornell University

Recent Advances in Automated Fact Checking Immanuel Trummer Cornell University Automation & Fact Checking Automation & Fact Checking Lorem Ipsum ... Identifying Lorem Ipsum ... Check-Worthy Claims

1.5k views • 96 slides
Bounded Model Checking bmc Revision: 1.11 1 [BiereCimattiClarkeZhu99] uses SAT for model

Bounded Model Checking bmc Revision: 1.11 1 [BiereCimattiClarkeZhu99] uses SAT for model

Bounded Model Checking bmc Revision: 1.11 1 [BiereCimattiClarkeZhu99] uses SAT for model checking historically not the first symbolic model checking approach scales better than original BDD based techniques mostly incomplete in

521 views • 28 slides
CTL Chapter 6 Part 2 Overview Review CTL Model Checking CTL model Checking algorithms

CTL Chapter 6 Part 2 Overview Review CTL Model Checking CTL model Checking algorithms

CTL Chapter 6 Part 2 Overview Review CTL Model Checking CTL model Checking algorithms for ( U ) Counter Examples and witnesses Symbolic Model Checking (Thursday) Binary Decision Trees

740 views • 40 slides
High-Dimensional Graphical Model Selection Anima Anandkumar U.C. Irvine Joint work with Vincent

High-Dimensional Graphical Model Selection Anima Anandkumar U.C. Irvine Joint work with Vincent

High-Dimensional Graphical Model Selection Anima Anandkumar U.C. Irvine Joint work with Vincent Tan (U. Wisc.) and Alan Willsky (MIT). Graphical Models: Definition Conditional Independence Dodgers Everton X A X B | X S B Red Sox S

905 views • 51 slides
Cure My FEVER: Building, Breaking, and Fixing Models for Fact-Checking Christopher Hidey Tuhin

Cure My FEVER: Building, Breaking, and Fixing Models for Fact-Checking Christopher Hidey Tuhin

Cure My FEVER: Building, Breaking, and Fixing Models for Fact-Checking Christopher Hidey Tuhin Chakrabarty Tariq Alhindi Siddharth Varia Kriste Krstovski Mona Diab Smaranda Muresan Automated Fact-checking and Related Tasks Source

382 views • 36 slides
Scalable Multi-Core Model Checking Alfons Laarman ( alfons@laarman.com ), Theory joint work with

Scalable Multi-Core Model Checking Alfons Laarman ( alfons@laarman.com ), Theory joint work with

Scalable Multi-Core Model Checking Alfons Laarman ( alfons@laarman.com ), Theory joint work with Jaco van de Pol and Tom van Dijk . 1 / 19 Scalable Multi-Core Model Checking Multi-Core Model Checking Research questions Can model checking

725 views • 61 slides
Statistical Model Checking and Rare Events Paolo Zuliani Joint work with Edmund M. Clarke

Statistical Model Checking and Rare Events Paolo Zuliani Joint work with Edmund M. Clarke

Statistical Model Checking and Rare Events Paolo Zuliani Joint work with Edmund M. Clarke Computer Science Department, CMU Probabilistic Verification Verification of stochastic system models via statistical model checking Temporal

686 views • 56 slides
Hoare Logic and Model Checking Model Checking But perhaps theres a clever way of

Hoare Logic and Model Checking Model Checking But perhaps theres a clever way of

Hoare Logic and Model Checking Model Checking But perhaps theres a clever way of compiling one into the other? Both have very different models of time How do they compare? We have seen two widely used temporal logics Relative

510 views • 10 slides
Hoare Logic and Model Checking 1. You should be able to model simple systems in NuSMV, an LTL

Hoare Logic and Model Checking 1. You should be able to model simple systems in NuSMV, an LTL

Hoare Logic and Model Checking 1. You should be able to model simple systems in NuSMV, an LTL Other interesting books: Model Checking by Clarke, Grumberg, and Peled Principles of Model Checking by Baier and Katoen 3 Course

636 views • 11 slides
On the Decidability of Model-Checking Information Flow Properties Raghavendra K. R. Joint Work:

On the Decidability of Model-Checking Information Flow Properties Raghavendra K. R. Joint Work:

Introduction Noninteference BSPs Results Conclusion On the Decidability of Model-Checking Information Flow Properties Raghavendra K. R. Joint Work: Deepak DSouza, Janardhan Kulkarni, Barbara Sprick, Raveendra Holla Indian Institute of

717 views • 26 slides
Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and Extensible Model Checking Framework a Modular and Extensible Model Checking Framework 3rd Estonian Summer School in Computer and System Science

628 views • 34 slides
Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and Extensible Model Checking Framework a Modular and Extensible Model Checking Framework 3rd Estonian Summer School in Computer and System Science

301 views • 8 slides
Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and

Software Model Checking Using Bogor Software Model Checking Using Bogor a Modular and Extensible Model Checking Framework a Modular and Extensible Model Checking Framework 3rd Estonian Summer School in Computer and System Science

777 views • 31 slides
Modeling and Analyzing Concurrent Systems using Model Checking Robert B. France 1 What is

Modeling and Analyzing Concurrent Systems using Model Checking Robert B. France 1 What is

Modeling and Analyzing Concurrent Systems using Model Checking Robert B. France 1 What is Model Checking? Model checking is an automated technique that, given a finite-state model of a system and a logical property , systematically

484 views • 36 slides
Model Checking: the Interval Way Alberto Molinari ( j.w. with L. Bozzelli, A. Montanari, A. Peron,

Model Checking: the Interval Way Alberto Molinari ( j.w. with L. Bozzelli, A. Montanari, A. Peron,

Model Checking: the Interval Way Alberto Molinari ( j.w. with L. Bozzelli, A. Montanari, A. Peron, P. Sala ) University of Udine Department of Mathematics, Computer Science, and Physics (DMIF) July 27, 2018 MODEL CHECKING Model checking : the

701 views • 38 slides
Automated Reasoning LTL Model Checking Alan Bundy Automated Reasoning LTL Model Checking

Automated Reasoning LTL Model Checking Alan Bundy Automated Reasoning LTL Model Checking

Automated Reasoning LTL Model Checking Alan Bundy Automated Reasoning LTL Model Checking Lecture 9, page 1 Introduction So far we have looked at theorem proving Powerful, especially where good sets of rewrite rules or decision procedures

639 views • 26 slides
Hoare Logic and Model Checking Model Checking Lecture 7: Introduction and background Dominic

Hoare Logic and Model Checking Model Checking Lecture 7: Introduction and background Dominic

Hoare Logic and Model Checking Model Checking Lecture 7: Introduction and background Dominic Mulligan Based on previous slides by Alan Mycroft and Mike Gordon Programming, Logic, and Semantics Group, University of Cambridge Academic year

972 views • 41 slides
Hoare Logic and Model Checking Model Checking See NuSMV homepage to download: http://nusmv.fbk.eu/

Hoare Logic and Model Checking Model Checking See NuSMV homepage to download: http://nusmv.fbk.eu/

Hoare Logic and Model Checking Model Checking See NuSMV homepage to download: http://nusmv.fbk.eu/ Was popular in semiconductor industry via Cadence SMV Could handle large models A re-implementation of the SMV model checker: Good

110 views • 10 slides
Advanced SAT-Techniques for Bounded Model Checking of Blackbox Designs Marc Herbstritt (joint

Advanced SAT-Techniques for Bounded Model Checking of Blackbox Designs Marc Herbstritt (joint

Advanced SAT-Techniques for Bounded Model Checking of Blackbox Designs Marc Herbstritt (joint work with Bernd Becker and Christoph Scholl) Institute of Computer Science Albert-Ludwigs-University Freiburg im Breisgau, Germany Presentation at

997 views • 54 slides

More recommend