Some thoughts on safety of machine learning Fabio Roli University - PowerPoint PPT Presentation

Pattern Recognition and Applications Lab Some thoughts on safety of machine learning Fabio Roli University of Cagliari, Italy HUML 2016, Venice, December 16th, 2016 Department of Electrical and Electronic Engineering

The black cloud… 1915 - 2001 http://pralab.diee.unica.it 2

The black cloud The novel tells the arrival of an enormous cloud of gas that appears to destroy the life on Earth by blocking the Sun's radiation. The motion of the cloud doesn’t follow physical laws , so one scientist argues that it might be a l ife-form with a degree of intelligence . Supposing that the cloud might be intelligent, the scientists try to communicate with it But human paradigms for information communication do not work ! “Could I put in this way? Said Kingsley. Between two absolutely identical individuals, no communication at all would be necessary because each individual would automatically know the experience of the other. …..Between two widely different individuals a vastly more complicated communication system is required” - F. Hoyle, The black cloud, 1957 - http://pralab.diee.unica.it 3

First issue http://pralab.diee.unica.it 4

Is antropomorphism good ? Is antropomorphism good for the human use of machine learning ? o Should we fabricate robots with human appearance ? o Should we be disappointed if algorithms make errors that humans wouldn’t have never done? “But if cattle and horses and lions had hands or could paint with their hands and create works such as men do, horses like horses and cattle like cattle also would depict the gods' shapes and make their bodies of such a sort as the form they themselves have ” Xenophánes; 570 – 480 BC http://pralab.diee.unica.it 5

Vision of a humanoid robot The iCub is the humanoid robot developed at the Italian Institute of Technology as part of the EU project RobotCub and subsequently adopted by more than 20 laboratories worldwide. It has 53 motors that move the head, arms and hands, waist, and legs. It can see and hear, it has the sense of proprioception (body configuration) and movement (using accelerometers and gyroscopes). [http://www.icub.org] The object recognition system of iCub uses visual features extracted with CNN models trained on the ImageNet dataset [G. Pasquale et al. MLIS 2015] http://pralab.diee.unica.it 6

iCub object recognition: example images [http://old.iit.it/projects/data-sets] http://pralab.diee.unica.it 7

The iCub object recognition pipeline Credits: Lorenzo Natale, Visual Learning of Objects and Tools on the iCub Robot, 2015 http://pralab.diee.unica.it 8

Generation of adversarial noise against iCub Disclaimer: unpublished work, work in progress at PRA Lab https://pralab.diee.unica.it The adversarial image x + r f(x) ≠ l is visually hard to distinguish from x Cup Plate Adversarial Noise 9 http://pralab.diee.unica.it 9

iCub is not a unique case … http://pralab.diee.unica.it 10

Black swans in ImageNet [Szegedy et al., Intriguing properties of neural networks, 2014] http://pralab.diee.unica.it 11

Adversarial faces [M. Sharif et al., ACM CCS 2016] M. Sharif et al. developed a systematic method to automatically generate attacks to face recognition systems, attacks which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the- art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. http://pralab.diee.unica.it 12

Adversarial images in mission-critical apps… [Patrick McDaniel et al., IEEE Security & Privacy, 2016] http://pralab.diee.unica.it 13

Adversarial images in mission-critical apps… [Patrick McDaniel et al., IEEE Security & Privacy, 2016] To humans, adversarial images are indistinguishable from original images. Left an ordinary image of a stop sign Right an image manipulated with adversarial noise and classified as a yield sign http://pralab.diee.unica.it 14

Big disappointment ? http://pralab.diee.unica.it 15

Low probability blind spots [M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015] 2 − class classification 5 Blue: legitimate class Red: illegal class 0 Spam/Ham for example − 5 − 5 0 5 Adversarial inputs in low probability regions can evade easily the classifier http://pralab.diee.unica.it 16

Low probability blind spots [M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015] 1.5C classification (MCS) 2 − class classification 5 5 0 0 − 5 − 5 − 5 0 5 − 5 0 5 Better enclosing legitimate data in feature space may improve classifier security http://pralab.diee.unica.it 17

Second issue http://pralab.diee.unica.it 18

Can machine learning be safe? M. Barreno et al., Can machine learning be secure? ASIACCS '06 Proceedings of the 2006 ACM Symposium on Information, computer and communications security http://pralab.diee.unica.it 19

Android malware detection � About one billion of users of Android mobile operating system � Thousands of new Android malware samples every day http://pralab.diee.unica.it 20

DREBIN: Android malware detector D. Arp et al. DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket NDSS 2014 - The Network and Distributed System Security Symposium w 1 s ≥ th Malware w 2 w 3 s � th ... Benign w n s < th http://pralab.diee.unica.it 21

Attacking a linear classifier Attacker’s goal: evasion SEND_SMS CAMERA SEND_SMS weights weights READ_SMS READ_SMS Android.hardware.wifi Android.hardware.wifi The Attacker can evade easily the classifier by manipulating a few features if weights are sparse ! http://pralab.diee.unica.it 22

Securing linear classifiers [A. Demontis et al., S+SSPR 2016] We learn feature weights evenly-distributed by solving this optimization problem: There is a trade off between sparsity and security ! http://pralab.diee.unica.it 23

Third issue http://pralab.diee.unica.it 24

Interpretability comes at a price ? Supposed that sparsity can facilitate interpretability , there is a trade off between interpretability and security in machine learning ? A. Demontis et al. On security and sparsity of linear classifiers for adversarial settings S+SSPR 2016 http://pralab.diee.unica.it 25

Last issue People are disappointed, and worried, for such errors… Can we trust in algorithms as well as we trust in humans ? http://pralab.diee.unica.it 26

Trust in humans or machines ? Algorithms are biased, but also humans are as well… When should you trust in humans and when in algorithms? http://pralab.diee.unica.it 27

Biases of humans and machines A bat and a ball together cost $ 1.10 The bat costs $ 1.0 more than the ball How much does the ball cost ? http://pralab.diee.unica.it 28

Back to the future ? Maybe we should reconsider again the distinction between two operation modes, System 1 and System 2 ? The discussion of the early days of machine learning about symbolic and sub-symbolic processing ? CoCo @ NIPS 2016 - Cognitive Computation: Integrating Neural and Symbolic Approaches http://pralab.diee.unica.it 29

Thanks for listening ! Any questions ? Engineering isn't about perfect solutions; it's about doing the best you can with limited resources (Randy Pausch, 1960-2008) http://pralab.diee.unica.it 30