fast homomorphic evaluation of deep discretized neural
play

Fast Homomorphic Evaluation of Deep Discretized Neural Networks - PowerPoint PPT Presentation

Feb 23, 2024 •97 likes •938 views

Fast Homomorphic Evaluation of Deep Discretized Neural Networks Florian Bourse Michele Minelli Matthias Minihold Pascal Paillier ENS, CNRS, PSL Research University, INRIA (Work done while visiting CryptoExperts) CRYPTO 2018 UCSB, Santa

  1. Fast Homomorphic Evaluation of Deep Discretized Neural Networks Florian Bourse Michele Minelli Matthias Minihold Pascal Paillier ENS, CNRS, PSL Research University, INRIA (Work done while visiting CryptoExperts) CRYPTO 2018 – UCSB, Santa Barbara

  2. Enc Enc Machine Learning as a Service (MLaaS) Michele Minelli 2 / 16

  3. Enc Enc Machine Learning as a Service (MLaaS) x Michele Minelli 2 / 16

  4. Enc Enc Machine Learning as a Service (MLaaS) x M ( x ) Michele Minelli 2 / 16

  5. Enc Enc Machine Learning as a Service (MLaaS) x Alice’s privacy! M ( x ) Michele Minelli 2 / 16

  6. Enc Enc Machine Learning as a Service (MLaaS) Possible solution: FHE. Michele Minelli 2 / 16

  7. Enc Machine Learning as a Service (MLaaS) Enc ( x ) Possible solution: FHE. Michele Minelli 2 / 16

  8. Machine Learning as a Service (MLaaS) Enc ( x ) Enc ( M ( x )) Possible solution: FHE. Michele Minelli 2 / 16

  9. Machine Learning as a Service (MLaaS) Enc ( x ) Enc ( M ( x )) Possible solution: FHE. ✓ Privacy data is encrypted (both input and output) ✗ Efficiency main issue with FHE-based solutions Michele Minelli 2 / 16

  10. Machine Learning as a Service (MLaaS) Enc ( x ) Enc ( M ( x )) Possible solution: FHE. ✓ Privacy data is encrypted (both input and output) ✗ Efficiency main issue with FHE-based solutions Goal of this work: homomorphic evaluation of trained networks. Michele Minelli 2 / 16

  11. (Very quick) refresher on neural networks Input Hidden Output layer layers layer d . . . . . . . . . . . . . . . Michele Minelli 3 / 16

  12. (Very quick) refresher on neural networks Computation for every neuron: x 1 w 1 w 2 x 2 y Σ . . . . . . x i , w i , y ∈ R Michele Minelli 3 / 16

  13. (Very quick) refresher on neural networks Computation for every neuron: x 1 w 1 w 2 x 2 y Σ . . . . . . x i , w i , y ∈ R (∑ ) y = f w i x i , i where f is an activation function . Michele Minelli 3 / 16

  14. A specific use case We consider the problem of digit recognition . Michele Minelli 4 / 16

  15. A specific use case We consider the problem of digit recognition . 7 Michele Minelli 4 / 16

  16. A specific use case We consider the problem of digit recognition . 7 Michele Minelli 4 / 16

  17. A specific use case We consider the problem of digit recognition . 7 Dataset: MNIST ( 60 000 training img + 10 000 test img). Michele Minelli 4 / 16

  18. % = = = State of the art Cryptonets [DGBL + 16 ] Michele Minelli 5 / 16

  19. % = = = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification Michele Minelli 5 / 16

  20. = = = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification ✓ Near state-of-the-art accuracy ( 98 . 95 % ) Michele Minelli 5 / 16

  21. = = = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification ✓ Near state-of-the-art accuracy ( 98 . 95 % ) ✗ Replaces sigmoidal activ. functions with low-degree f ( x ) = x 2 Michele Minelli 5 / 16

  22. = = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification ✓ Near state-of-the-art accuracy ( 98 . 95 % ) ✗ Replaces sigmoidal activ. functions with low-degree f ( x ) = x 2 ✗ Uses SHE = ⇒ parameters have to be chosen at setup time Michele Minelli 5 / 16

  23. = = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification ✓ Near state-of-the-art accuracy ( 98 . 95 % ) ✗ Replaces sigmoidal activ. functions with low-degree f ( x ) = x 2 ✗ Uses SHE = ⇒ parameters have to be chosen at setup time Main limitation The computation at neuron level depends on the total multiplicative depth of the network ⇒ bad for deep networks! Michele Minelli 5 / 16

  24. = State of the art Cryptonets [DGBL + 16 ] ✓ Achieves blind, non-interactive classification ✓ Near state-of-the-art accuracy ( 98 . 95 % ) ✗ Replaces sigmoidal activ. functions with low-degree f ( x ) = x 2 ✗ Uses SHE = ⇒ parameters have to be chosen at setup time Main limitation The computation at neuron level depends on the total multiplicative depth of the network ⇒ bad for deep networks! Goal: make the computation scale-invariant = ⇒ bootstrapping. Michele Minelli 5 / 16

  25. Enc Enc Enc = A restriction on the model We want to homomorphically compute the multisum ∑ w i x i i Michele Minelli 6 / 16

  26. = A restriction on the model We want to homomorphically compute the multisum ∑ w i x i i Given w 1 , . . . , w p and Enc ( x 1 ) , . . . , Enc ( x p ) , do ∑ w i · Enc ( x i ) i Michele Minelli 6 / 16

  27. = A restriction on the model We want to homomorphically compute the multisum ∑ w i x i i Given w 1 , . . . , w p and Enc ( x 1 ) , . . . , Enc ( x p ) , do ∑ w i · Enc ( x i ) i Proceed with caution In order to maintain correctness, we need w i ∈ Z Michele Minelli 6 / 16

  28. A restriction on the model We want to homomorphically compute the multisum ∑ w i x i i Given w 1 , . . . , w p and Enc ( x 1 ) , . . . , Enc ( x p ) , do ∑ w i · Enc ( x i ) i Proceed with caution In order to maintain correctness, we need w i ∈ Z = ⇒ trade-off efficiency vs. accuracy! Michele Minelli 6 / 16

  29. Discretized neural networks (DiNNs) Goal: FHE-friendly model of neural network. Michele Minelli 7 / 16

  30. Discretized neural networks (DiNNs) Goal: FHE-friendly model of neural network. Definition A DiNN is a neural network whose inputs are integer values in {− I , . . . , I } , and whose weights are integer values in {− W , . . . , W } , for some I , W ∈ N . For every activated neuron of the network, the activation function maps the multisum to integer values in {− I , . . . , I } . Michele Minelli 7 / 16

  31. Discretized neural networks (DiNNs) Goal: FHE-friendly model of neural network. Definition A DiNN is a neural network whose inputs are integer values in {− I , . . . , I } , and whose weights are integer values in {− W , . . . , W } , for some I , W ∈ N . For every activated neuron of the network, the activation function maps the multisum to integer values in {− I , . . . , I } . Not as restrictive as it seems: e.g., binarized NNs; Michele Minelli 7 / 16

  32. Discretized neural networks (DiNNs) Goal: FHE-friendly model of neural network. Definition A DiNN is a neural network whose inputs are integer values in {− I , . . . , I } , and whose weights are integer values in {− W , . . . , W } , for some I , W ∈ N . For every activated neuron of the network, the activation function maps the multisum to integer values in {− I , . . . , I } . Not as restrictive as it seems: e.g., binarized NNs; Trade-off between size and performance; Michele Minelli 7 / 16

  33. Discretized neural networks (DiNNs) Goal: FHE-friendly model of neural network. Definition A DiNN is a neural network whose inputs are integer values in {− I , . . . , I } , and whose weights are integer values in {− W , . . . , W } , for some I , W ∈ N . For every activated neuron of the network, the activation function maps the multisum to integer values in {− I , . . . , I } . Not as restrictive as it seems: e.g., binarized NNs; Trade-off between size and performance; (A basic) conversion is extremely easy. Michele Minelli 7 / 16

  34. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme (∑ ) ∑ w i · Enc ( x i ) = Enc w i x i i i Michele Minelli 8 / 16

  35. Enc Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function ( (∑ )) f w i x i i Michele Minelli 8 / 16

  36. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function 3 Bootstrap: can be costly ( (∑ )) Enc ∗ f w i x i i Michele Minelli 8 / 16

  37. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function 3 Bootstrap: can be costly 4 Repeat for all the layers ( (∑ )) Enc ∗ f w i x i i Michele Minelli 8 / 16

  38. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function 3 Bootstrap: can be costly 4 Repeat for all the layers Issues: Choose the message space: guess, statistics, or worst-case Michele Minelli 8 / 16

  39. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function 3 Bootstrap: can be costly 4 Repeat for all the layers Issues: Choose the message space: guess, statistics, or worst-case The noise grows: need to start from a very small noise Michele Minelli 8 / 16

  40. Homomorphic evaluation of a DiNN 1 Evaluate the multisum: easy – just need a linearly hom. scheme 2 Apply the activation function: depends on the function 3 Bootstrap: can be costly 4 Repeat for all the layers Issues: Choose the message space: guess, statistics, or worst-case The noise grows: need to start from a very small noise How do we apply the activation function homomorphically? Michele Minelli 8 / 16

Recommend

BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks ICPR 2016 Surat

BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks ICPR 2016 Surat

BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks ICPR 2016 Surat Teerapittayanon Brad McDanel H. T. Kung Harvard John A. Paulson School of Engineering and Applied Sciences 1 outline Motivation and Background Trend

1.49k views • 23 slides
Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks

Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks

Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks Charith Mendis Alex Renda Saman Amarasinghe Michael Carbin Compilers need to search through code Sequences High-level code Optimizing

1.08k views • 20 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Some Consequences about Oblivious Polynomial Evaluation from Existence of the Homomorphic and Non

Some Consequences about Oblivious Polynomial Evaluation from Existence of the Homomorphic and Non

Some Consequences about Oblivious Polynomial Evaluation from Existence of the Homomorphic and Non Committing Encryption Chunhua Su*, Tadashi Araragi $ , Takashi Nishide *, Kouichi Sakurai* *Department of Computer Science and Communication

260 views • 6 slides
Riptide: Fast End-to-End Binarized Neural Networks Josh Fromm, Meghan Cowan, Matthai Philipose,

Riptide: Fast End-to-End Binarized Neural Networks Josh Fromm, Meghan Cowan, Matthai Philipose,

Riptide: Fast End-to-End Binarized Neural Networks Josh Fromm, Meghan Cowan, Matthai Philipose, Luis Ceze, and Shwetak Patel 2 Canziani et al., An analysis of deep neural network models for practical applications. 2016 1-bit Matrix

2.33k views • 47 slides
and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

Band-limited Training and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast Training of Convolutional Networks through FFTs Fast Convolutional Nets With fbfft: A GPU Performance Evaluation Data: x

648 views • 46 slides
Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye S9554 - Fast Training of Deep Neural

Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye S9554 - Fast Training of Deep Neural

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 2 Without humans,

872 views • 49 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs Eunji

JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs Eunji

JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs Eunji Jeong , Sungwoo Cho, Gyeong-In Yu, Joo Seong Jeong, Dong-Jin Shin, Byung-Gon Chun Demo 1 Introduction Challenge Solution Results Deep Neural

452 views • 21 slides
Homomorphic Matrix Computation & Application to Neural Networks Xiaoqian Jiang, Miran Kim

Homomorphic Matrix Computation & Application to Neural Networks Xiaoqian Jiang, Miran Kim

Homomorphic Matrix Computation & Application to Neural Networks Xiaoqian Jiang, Miran Kim (University of Texas, Health Science Center at Houston) Kristin Lauter (Microsoft Research), Yongsoo Song (University of California, San Diego)

520 views • 28 slides
The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural

The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural

Deep Learning Antecedents Vision Case Study The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural Nets Deep Neural Nets Application ConvNets Jon Krohn LSTMs untapt jon@untapt.com

1.08k views • 106 slides
Homomorphic Evaluation of the AES Circuit Craig Gentry, Shai Halevi, Nigel P . Smart IBM

Homomorphic Evaluation of the AES Circuit Craig Gentry, Shai Halevi, Nigel P . Smart IBM

Homomorphic Evaluation of the AES Circuit Craig Gentry, Shai Halevi, Nigel P . Smart IBM Research and University Of Bristol. August 22, 2012 Craig Gentry, Shai Halevi, Nigel P . Smart Homomorphic Evaluation of the AES Circuit Slide 1

592 views • 30 slides
Laplacian Pyramid-based Complex Neural Network Learning for Fast MR Imaging Haoyun Liang Paul C.

Laplacian Pyramid-based Complex Neural Network Learning for Fast MR Imaging Haoyun Liang Paul C.

Laplacian Pyramid-based Complex Neural Network Learning for Fast MR Imaging Haoyun Liang Paul C. Lauterbur Research Center for Biomedical Imaging, SIAT, CAS Accelerating MRI Fast Imaging Sequence Parallel Imaging Compressed Sensing Deep

836 views • 8 slides
Deep Neural Nets and Features Sung-Eui Yoon ( ) Course URL:

Deep Neural Nets and Features Sung-Eui Yoon ( ) Course URL:

CS688: Web-Scale Image Search Deep Neural Nets and Features Sung-Eui Yoon ( ) Course URL: http://sgvr.kaist.ac.kr/~sungeui/IR Class Objectives Browse main components of deep neural nets Does not aim for giving in-depth

609 views • 32 slides
Homomorphic Encryption Lecture 18 And some applications Homomorphic Encryption Homomorphic

Homomorphic Encryption Lecture 18 And some applications Homomorphic Encryption Homomorphic

Homomorphic Encryption Lecture 18 And some applications Homomorphic Encryption Homomorphic Encryption Group Homomorphism: Two groups G and G are homomorphic if there exists a function (homomorphism) f:G G such that for all x,y G,

1.79k views • 165 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond .

From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond .

Why Traditional . . . How the Need for Fast . . . Faster Differentiation: . . . Beyond Traditional NN From Traditional Neural From Traditional NN . . . Networks to Deep Learning Need to Go Beyond . . . Constraints Are . . . and Beyond

548 views • 38 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Modular Hardware Architecture for Somewhat Homomorphic Function Evaluation CHES 2015 Sujoy Sinha

Modular Hardware Architecture for Somewhat Homomorphic Function Evaluation CHES 2015 Sujoy Sinha

1 Modular Hardware Architecture for Somewhat Homomorphic Function Evaluation CHES 2015 Sujoy Sinha Roy 1 , Kimmo Jrvinen 1 , Frederik Vercauteren 1 , Vassil Dimitrov 2 , and Ingrid Verbauwhede 1 1 ESAT/COSIC and iMinds, KU Leuven 2 The

849 views • 53 slides

More recommend