L 2 Reg u lari z ation Techniq u e u sing Keras IN TR OD U C TION - PowerPoint PPT Presentation

L 2 Reg u lari z ation Techniq u e u sing Keras IN TR OD U C TION TO TE N SOR FL OW IN R Colleen Bobbie Instr u ctor

The o v erfitting challenge Training data : Testing Data : Small Variance ( Good !) Large Variance ( Bad !) INTRODUCTION TO TENSORFLOW IN R

O v ercoming o v erfitting To tackle an o v er � t model : 1. Decrease o v er � � ing b y increasing training 2. Decrease o v er � � ing b y changing the comple x it y of the net w ork change net w ork str u ct u re ( n u mber of w eights ) change net w ork parameters ( v al u es of w eights ) These techniq u es are kno w n as reg u lari z ation . INTRODUCTION TO TENSORFLOW IN R

L 2 Reg u lari z ation (" Ridge Regression ") L 2 Reg u lari z ation : Res u lt : aims to � nd a model that ma y not � t the training data as w ell , b u t has the � e x ibilit y to � t other datasets small amo u nt of bias = signi � cant drop in v ariance in testing data 1 h � ps :// de v elopers . google . com / machine - learning / crash - co u rse / reg u lari z ation - for - simplicit y/ l 2- reg u lari z ation INTRODUCTION TO TENSORFLOW IN R

L 2 Reg u lari z ation in Keras In Keras : added to the model w hen la y ers are declared model <- keras_model_sequential() model %>% layer_dense(units = 15, activation = 'relu', input_shape = 8, kernel_regularizer = regularizer_l2(l = 0.001)) final loss = total loss + 0.001 * weight coefficient value INTRODUCTION TO TENSORFLOW IN R

Let ' s practice ! IN TR OD U C TION TO TE N SOR FL OW IN R

Dropo u t techniq u e u sing TFEstimators IN TR OD U C TION TO TE N SOR FL OW IN R Colleen Bobbie Instr u ctor

Dropo u t one of the most pop u lar forms of reg u lari z ation modi � es the ne u ral net w ork directl y A f u ll diagram ma y look like this : INTRODUCTION TO TENSORFLOW IN R

Dropo u t The � rst dropped diagram ma y look like this : INTRODUCTION TO TENSORFLOW IN R

Dropo u t Another diagram ma y look like this : INTRODUCTION TO TENSORFLOW IN R

Dropo u t in R Using the Estimators API w ith a dnn_classifier : ourmodel <- dnn_classifier( hidden_units = 6, feature_columns = ftr_colns, dropout = 0.5) dropo u t probabilit y is 0.5 or 50% the probabilit y of an y gi v en hidden la y er w ill be dropped is 50% INTRODUCTION TO TENSORFLOW IN R

Let ' s practice ! IN TR OD U C TION TO TE N SOR FL OW IN R

H y perparameter t u ning w ith tfr u ns IN TR OD U C TION TO TE N SOR FL OW IN R Colleen Bobbie Instr u ctor

H y perparameters for ne u ral net w orks n u mber of la y ers la y er acti v ations batch si z es and more ! INTRODUCTION TO TENSORFLOW IN R

Introd u ction to tfr u ns Which dropo u t is best ? # Create your dnn_classifier model mymodel <- dnn_classifier(feature_columns = featcols, hidden_units = c(40, 60, 10), n_classes = 2, label_vocabulary = c("N", "Y"), dropout = 0.5) 0.5? 0.2? 0.3? 0.4? INTRODUCTION TO TENSORFLOW IN R

T u ning a r u n best practice : de � ne � ags for ke y parameters o u tside of so u rce code . 1. Create a training script . R � le script that contains all R code for the model Helpf u l to sa v e this in w orking director y 2. Identif y the � ags � ags de � ne w hat rates y o u w o u ld like to test on each parameter for e x ample : dropout = c(0.2,0.3,0.4) INTRODUCTION TO TENSORFLOW IN R

T u ning a r u n Dropo u t : Dropo u t and Acti v ation : runs <- tuning_run( runs <- tuning_run( "modelsourcecode.R", "modelsourcecode.R", flags = list( flags = list( dropout = c(0.2, 0.3, 0.4, 0.5) dropout = c(0.2, 0.3, 0.4, 0.5), ) activation = c("relu", "softmax") ) ) ) INTRODUCTION TO TENSORFLOW IN R

E v al u ating the r u n ( s ) If r u nning in interacti v e mode , TensorBoard w ill sho w u p . Other w ise : runs[order(runs$eval_accuracy, decreasing = TRUE), ] Data frame: 4 x 24 run_dir eval_accuracy eval_accuracy_baseline eval_auc eval_auc_prec_recall 3 runs/2019-09-29T21-02-35Z 0.9927 0.5418 0.9988 0.9986 2 runs/2019-09-29T21-03-29Z 0.9891 0.5855 0.9998 0.9998 1 runs/2019-09-29T21-04-12Z 0.9564 0.5127 0.9888 0.9835 4 runs/2019-09-29T21-01-42Z 0.9491 0.5673 0.9917 0.9881 # ... with 20 more columns: # steps_completed, metrics, script, start, end, completed, output, source_code, context, type INTRODUCTION TO TENSORFLOW IN R

E v al u ating the r u n ( s ) dropout = c(0.2, 0.3, 0.4, 0.5) Data frame: 4 x 24 run_dir eval_accuracy eval_accuracy_baseline eval_auc eval_auc_precision_recall 3 runs/2019-09-29T21-02-35Z 0.9927 0.5418 0.9988 0.9986 2 runs/2019-09-29T21-03-29Z 0.9891 0.5855 0.9998 0.9998 1 runs/2019-09-29T21-04-12Z 0.9564 0.5127 0.9888 0.9835 4 runs/2019-09-29T21-01-42Z 0.9491 0.5673 0.9917 0.9881 # ... with 20 more columns: # steps_completed, metrics, script, start, end, completed, output, source_code, context, type Dropo u t = 0.4! INTRODUCTION TO TENSORFLOW IN R

Let ' s practice ! IN TR OD U C TION TO TE N SOR FL OW IN R

So long and thanks for all the fish IN TR OD U C TION TO TE N SOR FL OW IN R Colleen Bobbie Instr u ctor , signing o �

What y o u'v e learned Chapter 1: Introd u ction to TensorFlo w Chapter 2: Learning the basics core concepts linear regression model u sing the Core API TensorFlo w s y nta x linear regression model u sing Estimators TensorBoard Chapter 3: Deep learning in TensorFlo w Chapter 4: Model reg u lari z ation end - to - end w ork � o w of a Keras DNN L 2 reg u lari z ation (" Ridge Regression ") canned DNN u sing Estimators Dropo u t H y perparameter t u ning INTRODUCTION TO TENSORFLOW IN R

Learning on y o u r o w n Some ideas : 1. Other canned TFEstimator models s u ch as dnn_linear_combined_regressor 2. Te x t classi � cation model u sing Keras , incorporating a reg u lari z ation techniq u e 3. What q u estion ma � ers to y o u? Some reso u rces : F u ll RSt u dio TensorFlo w Doc u mentation Kaggle , Fi v eThirt y Eight INTRODUCTION TO TENSORFLOW IN R

Congrat u lations ! IN TR OD U C TION TO TE N SOR FL OW IN R