What deep generative models can do for you: Opportunities, - - PowerPoint PPT Presentation

What deep generative models can do for you: Opportunities, challenges, and open questions

1

Giulia Fanti Carnegie Mellon University

2

Vyas Sekar Chen Wang Alankar Jain Zinan Lin Hao Liang Kiran Thekumparampil Sewoong Oh

Common ML tools in networking

3

Classification

Classifying network traffic

Reinforcement learning

Traffic engineering

Unsupervised methods

Clustering signals

This talk: Generative models

4

• What are generative models?
• Why are they relevant now?
• How can they be useful in networking?
• What are the limitations?

What is a generative model?

• Models the joint probability distribution 𝑞(𝑦) of a dataset
• Example:

5

𝑦 𝑢 = 𝑔 𝑦 0: 𝑢 − 1 , 𝜄 + 𝑜[𝑢]

Noise Learned parameters Model How do we pick 𝑔? How to combine noise? Time t 𝑦[𝑢 − 1] 𝑦[0]

How are they used in the networking community?

Use dom domain n kno nowledg edge to extract high-level insights Design par param ametric model to model those insights Use da data to populate parameters

Network traffic has temporal patterns

𝑦 𝑢 = sin 𝜄𝑢 + 𝑜[𝑢]

! " = 1 day

Melamed (1993), Denneulin et al (2004), Swing, BURSE, Hierarchical bundling, Di et al (2014), …

Problems with this approach

7

• Requires new design for

every type of data Poor flexibility

• Doesn’t capture properties

that were not explicitly modeled Poor fidelity

Deep generative models

8

Design neur eural networ

• rk

to produce data of the right dimensionality Use da data to populate parameters

𝜄 ∈ 𝑆!

Generative Adversarial Networks (GANs): Breakthrough in generative modeling

• Prior approaches
• Likelihood-based
• Heavily rely on domain

knowledge

• GANs
• Adversarial learning
• Limited a priori assumptions

9

Generative Adversarial Networks (GANs)

Generator G Noise z Discriminator D FAKE! REAL

10

How can we use these tools in networking?

• Sharing synthetic data
• Discovering malicious inputs to black-box systems
• Understanding complex datasets

11

Sharing synthetic data

Use case 1

12

Vyas Sekar Chen Wang Alankar Jain Zinan Lin

github.com/fjxmlzn/DoppelGANger

Key stumbling block: Access to data

Enterprises Researchers

13

Unreproducible research Limited potential Collaborative opportunities go untapped Division A Division B

(Not a new) idea: Synthetic data models

14

Generative Model Generative Model Generative Model

Data Clearinghouse (ISAC, ISAO) Enterprises Researchers

Two main problems

Fideli lity Pri rivacy

Real Generated Generative Model Business secrets User data

15

Existing methods

16

Fi Fide delity Pr Privacy

Anonymized, raw data Expert-designed parametric models Machine-learned models DoppelGANger

Generating synthetic time series data with GANs

What kinds of data are we interested in?

17

Multi- dimensional time series With metadata (U.S., mobile traffic)

Datasets: Networking, security, and systems

• Cluster traces
• Go

Google: task resource usage logs from 12.k machines (2011)

• IB

IBM: resource usage measurements from 100k containers

• Traffic measurements
• Wi

Wikipedia web traffic: # daily views of Wikipedia articles (2016)

• FC

FCC Meas asuring Broad adban and America ca: Internet traffic and performance measurements from consumer devices around the country

18

DoppelGANger: Time series generation

19

RN RNN Noi Noise Au Auxiliary Di Discrimina nator

1: 1: re real 0: 0: fa fake

 Di Discrimina nator

1: 1: re real 0: 0: fa fake

R1,… ,…,R ,RS RN RNN Noi Noise RT-s+

s+1,…

,…,R ,RT

… …

Mi Min/Ma Max Gener Generator (M (MLP) (mi min±ma max/2 /2 Me Metadata Gener Generator (M (MLP) (A (A1, …, …, Am) Noi Noise

Part I: RNN + Batched Generation

20

Unbatched Batched

Challenge: Training on high-dynamic-range time series

21

Day

Part II: Auto normalization

• Standard normalization: Normalize by gl

global min/max

• DoppelGANger: Normalize each timeseries individually
• Store min/max as “fake” metadata

22

max min (min, max) (min, max) (min, max)

23

Challenge: Complex relationships in metadata

24

• Need to capture relation between metadata and time series
• E.g., Cable vs Mobile users
• Straw man: Joint generator of metadata and time series
• Problem: too hard for a single generator

Time series min value

Count

Before: Single generator

Part III: Decoupled Generation, Auxiliary Discriminator

• Two stage decoupling
• Generate metadata (using a standard MLP)
• Generate measurements conditioned on metadata
• Auxiliary discriminator for metadata alone

25

Histogram of !"#$!%&

'

per time series

26

Withou hout auxiliary discriminator Wi With auxiliary discriminator Count Count Time series min value

Putting it together

27

RN RNN Noi Noise Au Auxiliary Di Discrimina nator

1: 1: re real 0: 0: fa fake

 Di Discrimina nator

1: 1: re real 0: 0: fa fake

R1,… ,…,R ,RS RN RNN Noi Noise RT-s+

s+1,…

,…,R ,RT

… …

Mi Min/Ma Max Gener Generator (M (MLP) (mi min±ma max/2 /2 Me Metadata Gener Generator (M (MLP) (A (A1, …, …, Am) Noi Noise

Temporal Correlations

Microbenchmark

28

Predicting job failures in a compute cluster

Downstream task

• Train on synthetic, test on real

29

Evaluating privacy

• Protecting business secrets
• Aggregate functions of the data
• User privacy
• Differential privacy
• Robustness against membership inference

30

Differentially-private SGD kills fidelity in GANs

31

Open questions: Synthetic data generation

• Fi

Fidelity

• Long sequences of data
• Stateful protocols
• Pr

Privacy

• Differentially-private GANs
• New privacy metrics?

32

Identifying malicious inputs to black-box systems

Use case 2

33

Vyas Sekar Zinan Lin Hao Liang

Bl Black-bo box De Devi vices es and System ems Abound

IoT Devices Servers / Routers Control Units in Vehicles / Manufacturing NO NO so source co code / bi bina nary / pr protoco col fo format / de design do doc

Towards Oblivious Network Analysis using GANs HotNets'19 11/14/2019 34

Id Identifying At Attack Pa Packets is is Ha Hard

Towards Oblivious Network Analysis using GANs HotNets'19 11/14/2019

Send packets We We wa want to to id identif tify at attack ack pack packets, but but do do NO NOT ha have so source co code or system descr cript ption

35

Attacker

Motivating example

• Packet classification
• Vamanan et al [SIGCOMM 2010]
• Singh et al [SIGCOMM 2013]
• Yingchareonthawornchai et al [TON 2018]
• Liang et al [SIGCOMM 2019]
• Rashelbach et al [SIGCOMM 2020]
• Many more…

36

Classification Time Can an attacker identify many y packets with hi high classification times?

Random packet generation

• NeuroCuts, Liang et al [SIGCOMM 2019]

37

Thre Threshold hold Slo low p packets Fa Fast pac packets Can can we generate many, d , diverse s slo low packets? Classification Time (ms) Number of packets 2,000 total packets

Common approaches

• Fuzzing tools
• Random sampling
• Optimization of black-box functions
• Bayesian optimization
• Genetic algorithms
• Simulated annealing

38

GANs can help!

11/14/2019 Towards Oblivious Network Analysis using GANs HotNets'19

Classification decision tree Random Packets GAN Training Dataset “Fast” packets “Slow” packets

Ap Approa

• ach 1:

1: Va Vanilla GA GAN

1% 1%

16

• Challenge: too little training data

11/14/2019 Towards Oblivious Network Analysis using GANs HotNets'19

GAN Training Dataset “Fast” packets “Slow” packets Generate packets with condition=“slow”

Am AmpGAN AN: Tr Training with Feedback

18

Random Packets Classification decision tree

Am AmpGAN AN

Results

41

Classification Time (ms) Number of packets Number of packets Random

• m

pac packets Am AmpGAN AN Thre Threshold hold

Results

42

System Calls Fraction of “slow” packets

AmpGAN

Genetic Algorithms Simulated Annealing Generalized SA Bayesian Optimization AmpGAN

2.5x jump 10x jump

Open questions

• Sequences of inputs
• Can we use this to optimize systems as well as finding attacks
• E.g., CherryPick [NSDI 2017]

43

Extracting insights from unstructured data

Use case 3

44

Zinan Lin Kiran Thekumparampil Sewoong Oh

github.com/fjxmlzn/InfoGAN-CR

• How do 𝑨,s control the factors?

Disentangled GANs

Generator 𝑨$ 𝑨% 𝑨&

…

𝑙 factors

• Hair color
• Rotation
• Background
• Bangs

Vanilla GANs 𝑨$ 𝑨% 𝑨&

…

Factor$ Factor% Factor'

…

45 Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR

𝑒 input noise Disentangled GANs 𝑑$ 𝑑% 𝑑'

…

Factor$ Factor% Factor'

…

𝑨(s

Examples of Disentanglement

Generator 𝑑$ 𝑑% 𝑑&

…

Changing only: 𝑑$ 𝑑% 𝑑) 𝑑* 𝑑+ hair color rotation lighting background bangs shape scale rotation x-position y-position

(CelebA dataset) (dSprites dataset)

46

𝑨(s

* CelebA example is generated by InfoGAN-CR. Dsprites example is synthetic for illustration.

Latent codes The remaining noise dimensions

Our Solution: Contrastive Regularizer (CR)

Generator (𝐻)

𝑗 = 1 𝑗 = 2 𝑗 = 3 Same shape Same x-position Same y-position

47

Same i-th latent code

• Use two latent codes (𝑑-,…,𝑑,,…,𝑑.), (𝑑-

/,…,𝑑, /,…,𝑑. / ) to generate a pair of

images

Equa l

Intuition of Contrastive Regularizer (CR)

Generator (𝐻)

𝑗 = 1 𝑗 = 2 𝑗 = 3

Contrastive Regularizer (CR)

1 2 3

Classification task!

48

Same i-th latent code

• Use two latent codes (𝑑-,…,𝑑,,…,𝑑.), (𝑑-

/,…,𝑑, /,…,𝑑. / ) to generate a pair of

images

Equa l

InfoGAN-CR

InfoGAN-CR loss: min

0,2,3 max 4

𝑀567 𝐻, 𝐸 − 𝜇𝐽 𝐻, 𝑅 − 𝛽𝑀8(𝐻, 𝐼)

cl clas assificat cation accu accuracy acy of

• f CR

CR

𝐼

[1] InfoGAN. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (NeurIPS 2016)

! ∈ ℝ! $ ∈ ℝ" % ∈ ℝ#

!

GAN Discriminator InfoGAN Encoder

" #

ℝ" CR ℝ ̂ $ ∈ ℝ" Input Noise Latent Factors %′ ∈ ℝ# %′′ ∈ ℝ#

49

InfoGAN [1] GAN’s adversarial loss Mutual info loss

Key open question

• Can disentanglement help us make sense of networking data?
• Reverse-engineer protocols
• Categorize complex data patterns
• Has not been tried in networking domain
• Time series disentanglement
• Himberg, Hyvärinen, Esposito (2004)

50

Take-home messages

• Deep generative models show promise for networking applications
• Synthetic data generation
• Identifying malicious packets for black-box systems
• Extracting structural insights from data
• They often cannot be applied off the shelf
• New architectures
• Data pre-processing pipelines
• Training mechanisms

51