Training and u pdating models AD VAN C E D N L P W ITH SPAC Y - PowerPoint PPT Presentation

Training and u pdating models AD VAN C E D N L P W ITH SPAC Y Ines Montani spaC y core de v eloper

Wh y u pdating the model ? Be � er res u lts on y o u r speci � c domain Learn classi � cation schemes speci � call y for y o u r problem Essential for te x t classi � cation Ver y u sef u l for named entit y recognition Less critical for part - of - speech tagging and dependenc y parsing ADVANCED NLP WITH SPACY

Ho w training w orks (1) 1. Initiali z e the model w eights randoml y w ith nlp.begin_training 2. Predict a fe w e x amples w ith the c u rrent w eights b y calling nlp.update 3. Compare prediction w ith tr u e labels 4. Calc u late ho w to change w eights to impro v e predictions 5. Update w eights slightl y 6. Go back to 2. ADVANCED NLP WITH SPACY

Ho w training w orks (2) Training data : E x amples and their annotations . Te x t : The inp u t te x t the model sho u ld predict a label for . Label : The label the model sho u ld predict . Gradient : Ho w to change the w eights . ADVANCED NLP WITH SPACY

E x ample : Training the entit y recogni z er The entit y recogni z er tags w ords and phrases in conte x t Each token can onl y be part of one entit y E x amples need to come w ith conte x t ("iPhone X is coming", {'entities': [(0, 8, 'GADGET')]}) Te x ts w ith no entities are also important ("I need a new phone! Any tips?", {'entities': []}) Goal : teach the model to generali z e ADVANCED NLP WITH SPACY

The training data E x amples of w hat w e w ant the model to predict in conte x t Update an e x isting model : a fe w h u ndred to a fe w tho u sand e x amples Train a ne w categor y : a fe w tho u sand to a million e x amples spaC y' s English models : 2 million w ords Us u all y created man u all y b y h u man annotators Can be semi - a u tomated – for e x ample , u sing spaC y' s Matcher ! ADVANCED NLP WITH SPACY

Let ' s practice ! AD VAN C E D N L P W ITH SPAC Y

The training loop AD VAN C E D N L P W ITH SPAC Y Ines Montani spaC y core de v eloper

The steps of a training loop 1. Loop for a n u mber of times . 2. Sh u� e the training data . 3. Di v ide the data into batches . 4. Update the model for each batch . 5. Sa v e the u pdated model . ADVANCED NLP WITH SPACY

Recap : Ho w training w orks Training data : E x amples and their annotations . Te x t : The inp u t te x t the model sho u ld predict a label for . Label : The label the model sho u ld predict . Gradient : Ho w to change the w eights . ADVANCED NLP WITH SPACY

E x ample loop TRAINING_DATA = [ ("How to preorder the iPhone X", {'entities': [(20, 28, 'GADGET')]}) # And many more examples... ] # Loop for 10 iterations for i in range(10): # Shuffle the training data random.shuffle(TRAINING_DATA) # Create batches and iterate over them for batch in spacy.util.minibatch(TRAINING_DATA): # Split the batch in texts and annotations texts = [text for text, annotation in batch] annotations = [annotation for text, annotation in batch] # Update the model nlp.update(texts, annotations) # Save the model nlp.to_disk(path_to_model) ADVANCED NLP WITH SPACY

Updating an e x isting model Impro v e the predictions on ne w data Especiall y u sef u l to impro v e e x isting categories , like PERSON Also possible to add ne w categories Be caref u l and make s u re the model doesn ' t " forget " the old ones ADVANCED NLP WITH SPACY

Setting u p a ne w pipeline from scratch # Start with blank English model nlp = spacy.blank('en') # Create blank entity recognizer and add it to the pipeline ner = nlp.create_pipe('ner') nlp.add_pipe(ner) # Add a new label ner.add_label('GADGET') # Start the training nlp.begin_training() # Train for 10 iterations for itn in range(10): random.shuffle(examples) # Divide examples into batches for batch in spacy.util.minibatch(examples, size=2): texts = [text for text, annotation in batch] annotations = [annotation for text, annotation in batch] # Update the model nlp.update(texts, annotations) ADVANCED NLP WITH SPACY

Let ' s practice ! AD VAN C E D N L P W ITH SPAC Y

Best practices for training spaC y models AD VAN C E D N L P W ITH SPAC Y Ines Montani spaC y core de v eloper

Problem 1: Models can " forget " things E x isting model can o v er � t on ne w data e . g .: if y o u onl y u pdate it w ith WEBSITE , it can "u nlearn " w hat a PERSON is Also kno w n as " catastrophic forge � ing " problem ADVANCED NLP WITH SPACY

Sol u tion 1: Mi x in pre v io u sl y correct predictions For e x ample , if y o u' re training WEBSITE , also incl u de e x amples of PERSON R u n e x isting spaC y model o v er data and e x tract all other rele v ant entities BAD : TRAINING_DATA = [ ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]}) ] GOOD : TRAINING_DATA = [ ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]}), ('Obama is a person', {'entities': [(0, 5, 'PERSON')]}) ] ADVANCED NLP WITH SPACY

Problem 2: Models can ' t learn e v er y thing spaC y' s models make predictions based on local conte x t Model can str u ggle to learn if decision is di � c u lt to make based on conte x t Label scheme needs to be consistent and not too speci � c For e x ample : CLOTHING is be � er than ADULT_CLOTHING and CHILDRENS_CLOTHING ADVANCED NLP WITH SPACY

Sol u tion 2: Plan y o u r label scheme caref u ll y Pick categories that are re � ected in local conte x t More generic is be � er than too speci � c Use r u les to go from generic labels to speci � c categories BAD : LABELS = ['ADULT_SHOES', 'CHILDRENS_SHOES', 'BANDS_I_LIKE'] GOOD : LABELS = ['CLOTHING', 'BAND'] ADVANCED NLP WITH SPACY

Let ' s practice ! AD VAN C E D N L P W ITH SPAC Y

Wrapping u p AD VAN C E D N L P W ITH SPAC Y Ines Montani spaC y core de v eloper

Yo u r ne w spaC y skills E x tract ling u istic feat u res : part - of - speech tags , dependencies , named entities Work w ith pre - trained statistical models Find w ords and phrases u sing Matcher and PhraseMatcher match r u les Best practices for w orking w ith data str u ct u res Doc , Token Span , Vocab , Lexeme Find semantic similarities u sing w ord v ectors Write c u stom pipeline components w ith e x tension a � rib u tes Scale u p y o u r spaC y pipelines and make them fast Create training data for spaC y' statistical models Train and u pdate spaC y' s ne u ral net w ork models w ith ne w data ADVANCED NLP WITH SPACY

More things to do w ith spaC y (1) Training and u pdating other pipeline components Part - of - speech tagger Dependenc y parser Te x t classi � er ADVANCED NLP WITH SPACY

More things to do w ith spaC y (2) C u stomi z ing the tokeni z er Adding r u les and e x ceptions to split te x t di � erentl y Adding or impro v ing s u pport for other lang u ages 45+ lang u ages c u rrentl y Lots of room for impro v ement and more lang u ages Allo w s training models for other lang u ages ADVANCED NLP WITH SPACY

See the w ebsite for more info and doc u mentation ! spac y. io ADVANCED NLP WITH SPACY

Thanks and see y o u soon ! AD VAN C E D N L P W ITH SPAC Y