adversarial examples for evaluating
play

Adversarial Examples for Evaluating Reading Comprehension Systems - PowerPoint PPT Presentation

Feb 24, 2024 •338 likes •1.04k views

Adversarial Examples for Evaluating Reading Comprehension Systems Robin Jia and Percy Liang Stanford University Reading Comprehension Task Question: The number of new Huguenot colonists declined after what year? Paragraph: The largest

  1. Adversarial Examples for Evaluating Reading Comprehension Systems Robin Jia and Percy Liang Stanford University

  2. Reading Comprehension Task Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700 ; thereafter, the numbers declined…” Correct Answer: “ 1700 ” Stanford Question Answering Dataset (Rajpurkar et al., 2016) 2

  3. Progress on SQuAD Human Performance Do these models actually understand language? Logistic Regression Baseline SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ 3

  4. Adversarial Evaluation Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700 ; thereafter, the numbers declined. The number of old Acadian colonists declined after the year of 1675 .” Correct Answer: “ 1700 ” Predicted Answer: “ 1675 ” Model used: BiDAF Ensemble (Seo et al., 2016) 4

  5. Adversarial Evaluation Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700 ; thereafter, the numbers declined. expected yet later be basis need young only required 1961 .” Correct Answer: “ 1700 ” Predicted Answer: “ 1961 ” Model used: BiDAF Ensemble (Seo et al., 2016) 5

  6. Outline • Inspiration/Motivation • Adding Grammatical Sentences • Adding Word Salad • Trying to build better systems 6

  7. Outline • Inspiration/Motivation • Adding Grammatical Sentences • Adding Word Salad • Trying to build better systems 7

  8. Some Inspiration + .007 * = Panda Nematode Gibbon 58% confidence 8% confidence 99% confidence Local perturbations don’t change semantics of image, but models are oversensitive to small differences! Goodfellow et al., 2014. 8

  9. Local perturbations of text Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700 ; thereafter, the numbers amount declined decreased …” Plausible alternative answers not always present Hard to find a lot of perturbations to try Li et al., 2017 9

  10. Preserving Semantics • For images, most local perturbations preserve semantics • For text, most local perturbations alter semantics • Even changing one word by a small amount may not preserve semantics (e.g. entity names) 10

  11. Concatenative Adversaries • Instead of locally altering the input, append distracting text to the paragraph • Must ensure that added text does not actually answer the question 11

  12. Distracting Text Question: “The number of new Huguenot colonists declined after what year ?” Distracting text: “The number of new Huguenot colonists declined after the year 1675 .” Answer according to text: “ 1675 ” 12

  13. Distracting Text Question: “The number of new Huguenot colonists declined after what year ?” Distracting text: “The number of new old Huguenot Acadian colonists declined after the year 1675 .” Answer according to text: N/A Local perturbations change semantics of sentence, but models are overly stable/insensitive to these changes! 13

  14. Outline • Inspiration/Motivation • Adding Grammatical Sentences • Adding Word Salad • Trying to build better systems 14

  15. Grammatical Distractors What city did Tesla move to in 1880? Prague Change entities, Generate fake answer with numbers, antonyms same NER/POS tag What city did Tadakatsu move to in 1881? Chicago Convert to declarative sentence Tadakatsu moved the city of Chicago to in 1881. Have crowdworkers fix errors Tadakatsu moved to the city of Chicago in 1881. 15

  16. Four “ d ev” systems SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ *Some of our results are on older versions of models than shown here 16

  17. Results (4 “dev” systems) System Original AddOneSent BiDAF, ensemble (Seo et al., 2016) 80.0 46.9 BiDAF, single (Seo et al., 2016) 75.5 45.7 Match-LSTM, ensemble (Wang & Jiang, 2016) 75.4 41.8 Match-LSTM, single (Wang & Jiang, 2016) 71.4 39.0 Human Performance 92.6 89.2 17

  18. Picking a worst-case sentence Tadakatsu moved the city of Chicago to in 1881. Have crowdworkers fix errors Tadakatsu moved to the city of Chicago in 1881. Tadakatsu moved to Chicago in 1881. In 1881, Tadakatsu moved to the city of Chicago. Model failed if distracted by any of these 18

  19. Results (4 “dev” systems ) System Original AddOneSent AddSent BiDAF, ensemble (Seo et al., 2016) 80.0 46.9 34.2 BiDAF, single (Seo et al., 2016) 75.5 45.7 34.3 Match-LSTM, ensemble (Wang & Jiang, 2016) 75.4 41.8 29.4 Match-LSTM, single (Wang & Jiang, 2016) 71.4 39.0 27.3 Human Performance 92.6 89.2 79.5 19

  20. Computers on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph Chicago … Model 20

  21. Computers on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph Chicago … Model Deterministically choose argmax 21

  22. Humans on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph Chicago … Crowd Only get noisy samples! 22

  23. Humans on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph Chicago … Crowd Only get noisy samples! 23

  24. Humans on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph #2 Chicago … Crowd Only get noisy samples! 24

  25. Humans on AddSent What city did Tesla move to in 1880? Prague Adversarial Gospić Paragraph #3 Chicago … Crowd Noise augmented when picking worst-case sentence 25

  26. Twelve “test” systems SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ *Some of our results are on older versions of models than shown here 26

  27. Results (12 “test” systems) System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2 27

  28. Results (12 “test” systems) System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2 28

  29. Results (12 “test” systems) System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2 29

  30. Partial Matches Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700 ; thereafter, the numbers declined. The number of old Acadian colonists declined after the year of 1675 .” All models distracted by sentences with only partial match with the question 30

  31. Partial Matches Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689, in seven ships as part of the organised migration, but quite a few arrived as late as 1700 ; thereafter , the numbers declined , and only small groups arrived at a time.” Correct Answer: “ 1700 ” Stanford Question Answering Dataset (Rajpurkar et al., 2016) 31

  32. Outline • Inspiration/Motivation • Adding Grammatical Sentences • Adding Word Salad • Trying to build better systems 32

Recommend

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples Nicholas

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples Nicholas

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples Nicholas Carlini Google Research Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples Lessons Learned from Evaluating the

1.28k views • 127 slides
Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML

Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML

Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML models that an attacker has intentionally designed to cause the model to make a mistake 1 Why this is interesting: Safety. Interpretability.

881 views • 22 slides
Adversarial Examples and Adversarial Training Ian Goodfellow, Sta ff Research Scientist, Google

Adversarial Examples and Adversarial Training Ian Goodfellow, Sta ff Research Scientist, Google

Adversarial Examples and Adversarial Training Ian Goodfellow, Sta ff Research Scientist, Google Brain CS 231n, Stanford University, 2017-05-30 Overview What are adversarial examples? Why do they happen? How can they be used to

866 views • 43 slides
A Closer Look at Adversarial Examples for Separated Data Kamalika Chaudhuri University of

A Closer Look at Adversarial Examples for Separated Data Kamalika Chaudhuri University of

A Closer Look at Adversarial Examples for Separated Data Kamalika Chaudhuri University of California, San Diego Adversarial Examples Gibbon Panda Small perturbation to legitimate inputs causing misclassification Adversarial Examples Can

1.41k views • 42 slides
Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin

Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin

Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin Kwok Adversarial examples Adversarial examples Imperceptible perturbations to an input can change a neural network's prediction adversarial

1.06k views • 23 slides
Thermometer Encoding: One Hot Way to Resist Adversarial Examples Stanford, 2017-11-16 Aurko Roy*

Thermometer Encoding: One Hot Way to Resist Adversarial Examples Stanford, 2017-11-16 Aurko Roy*

Thermometer Encoding: One Hot Way to Resist Adversarial Examples Stanford, 2017-11-16 Aurko Roy* Colin Ra ff el Jacob Ian Buckman* Goodfellow *joint first author Adversarial Examples Adversarial Definitely Probably panda perturbation

607 views • 15 slides
Adversarial Examples in NLP Sameer Singh sameer@uci.edu @sameer_ sameersingh.org What are

Adversarial Examples in NLP Sameer Singh sameer@uci.edu @sameer_ sameersingh.org What are

Slides: http://tiny.cc/adversarial Adversarial Examples in NLP Sameer Singh sameer@uci.edu @sameer_ sameersingh.org What are Adversarial Examples? panda gibbon 57.7% confidence 99.3% confidence [Goodfellow et al, ICLR 2015 ]

1.56k views • 43 slides
? AdVersarial: Defeating Perceptual Ad Blocking with Adversarial Examples Florian Tramr

? AdVersarial: Defeating Perceptual Ad Blocking with Adversarial Examples Florian Tramr

Todays Story: How I got my first Death Threat ? AdVersarial: Defeating Perceptual Ad Blocking with Adversarial Examples Florian Tramr October 8 th 2019 Joint work with Pascal Dupr, Gili Rusak, Giancarlo Pellegrino and Dan Boneh The

415 views • 29 slides
Explaining and Harnessing Adversarial Examples Ian J. Goodfellow, Jonathon Shlens, &

Explaining and Harnessing Adversarial Examples Ian J. Goodfellow, Jonathon Shlens, &

Explaining and Harnessing Adversarial Examples Ian J. Goodfellow, Jonathon Shlens, & Christian Szegedy Presented by - Kawin Ethayarajh and Abhishek Tiwari Introduction - adversarial examples : Inputs formed by applying small but

1.49k views • 103 slides
Neglected topics CS 446 Adversarial examples and deep networks 1 / 23 Adversarial

Neglected topics CS 446 Adversarial examples and deep networks 1 / 23 Adversarial

Neglected topics CS 446 Adversarial examples and deep networks 1 / 23 Adversarial examples? Standard ML setup: We have training data; try to do well on withheld testing data. Adversarial/robust ML setup: We have training

672 views • 27 slides
Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Security

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Security

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Security Seminar, Stanford University, 2017-01-17 In this presentation Intriguing Properties of Neural Networks Szegedy et al, 2013

939 views • 44 slides
Physical Adversarial Examples Alex Kurakin Ian Goodfellow Output STOP Machine Learning

Physical Adversarial Examples Alex Kurakin Ian Goodfellow Output STOP Machine Learning

Physical Adversarial Examples Alex Kurakin Ian Goodfellow Output STOP Machine Learning Training Examples Hidden units / BICYCLE features CAR PEDESTRIA N Parameters Input ImageNet (Russakovsky et al 2015) Adversarial Examples: Images

369 views • 19 slides
On the Defense Against Adversarial Examples Beyond the Visible Spectrum Anthony Ortiz 1 , Olac

On the Defense Against Adversarial Examples Beyond the Visible Spectrum Anthony Ortiz 1 , Olac

On the Defense Against Adversarial Examples Beyond the Visible Spectrum Anthony Ortiz 1 , Olac Fuentes 1 , Dalton Rosario 2 , Christopher Kiekintveld 1 1 Department of Computer Science, UTEP 2 US Army Research Laboratory Adversarial Examples on

460 views • 19 slides
Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Guest

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Guest

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Guest lecture for CS 294-131, UC Berkeley, 2016-10-05 In this presentation Intriguing Properties of Neural Networks Szegedy et al, 2013

660 views • 44 slides
Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin

Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin

Synthesizing Robust Adversarial Examples Anish Athalye*, Logan Engstrom*, Andrew Ilyas*, Kevin Kwok Standard Adversarial Examples Given image x ; target class y Maximize with projected gradient descent: Standard Adversarial Examples Standard

560 views • 15 slides
Adversarial Examples and Adversarial Training Innova&ve Technology Leader program January 22

Adversarial Examples and Adversarial Training Innova&ve Technology Leader program January 22

Adversarial Examples and Adversarial Training Innova&ve Technology Leader program January 22 nd 2018 Florian Tramr Stanford Deep Learning is Super Smart! 2 Is it really? + . 007 = Im sure this Im certain this is a panda is

452 views • 12 slides
Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Presentation at San Francisco AI Meetup, 2016-08-18 In this presentation Intriguing Properties of Neural Networks Szegedy et al, 2013

690 views • 32 slides
Transferable Adversarial Examples: Insights, A9acks & Defenses June 12 th 2017 Florian

Transferable Adversarial Examples: Insights, A9acks & Defenses June 12 th 2017 Florian

Transferable Adversarial Examples: Insights, A9acks & Defenses June 12 th 2017 Florian Tramr Joint work with Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh & Patrick McDaniel Adversarial Examples Threat Model: White-Box

480 views • 27 slides
Vulnerability of machine learning models to adversarial examples Petra Vidnerov Institute of

Vulnerability of machine learning models to adversarial examples Petra Vidnerov Institute of

Vulnerability of machine learning models to adversarial examples Petra Vidnerov Institute of Computer Science The Czech Academy of Sciences Hora Informaticae 2016 Outline Introduction Works on adversarial examples Our work Genetic

855 views • 46 slides
The case for dynamic defenses against adversarial examples Ian Goodfellow SafeML ICLR Workshop

The case for dynamic defenses against adversarial examples Ian Goodfellow SafeML ICLR Workshop

The case for dynamic defenses against adversarial examples Ian Goodfellow SafeML ICLR Workshop 2019-05-06 New Orleans Based on https://arxiv.org/pdf/1903.06293.pdf Definition Adversarial examples are inputs to machine learning models that

412 views • 22 slides
Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist

Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Presentation at HORSE 2016 London, 2016-09-19 In this presentation Intriguing Properties of Neural Networks Szegedy et al, 2013 Explaining

497 views • 22 slides
Towards Robust Detection of Adversarial Examples Tianyu Pang, Chao Du, Yinpeng Dong

Towards Robust Detection of Adversarial Examples Tianyu Pang, Chao Du, Yinpeng Dong

Towards Robust Detection of Adversarial Examples Tianyu Pang, Chao Du, Yinpeng Dong and Jun Zhu Department of Computer Science and Technology Tsinghua University TSAIL NeurIPS | 2018 Adversarial Examples From

590 views • 10 slides
Confidence-Calibrated Adversarial Training Generalizing to Unseen Attacks David Stutz, Matthias

Confidence-Calibrated Adversarial Training Generalizing to Unseen Attacks David Stutz, Matthias

Confidence-Calibrated Adversarial Training Generalizing to Unseen Attacks David Stutz, Matthias Hein, Bernt Schiele 2-Minute Overview Problem: Robustness to various adversarial examples. Adversarial training on L adversarial examples:

635 views • 34 slides
Deep Adversarial Learning for NLP 9:00 10:30 Introduction and Adversarial Training, GANs

Deep Adversarial Learning for NLP 9:00 10:30 Introduction and Adversarial Training, GANs

Deep Adversarial Learning for NLP 9:00 10:30 Introduction and Adversarial Training, GANs William Wang 10:30 11:00 Break - 11:00 12:15 Adversarial Examples Sameer Singh 12:15 12:30 Conclusions and Question Answering William

1.75k views • 105 slides

More recommend