Migrating to chatbot -- Madhu Gopinathan -- Sanjay Mohan
MakeMyTrip: India’s One Stop Travel Shop Flights Hotels Holidays Bus Cabs Trains Gift Cards Experiences Visa Homes 40 million customers
Evolution of Customer Support Write to us IVR
Timeline of refund status Oct 21: I will sue you. Where is my refund? Oct 10: Rescheduled my flight Oct 25: Oct 18 – Night: directly with airline I am really pissed! Call me after 7 pm Oct 26: Oct 16 – Night: When Ok. Got it. Thanks will I get my refund? Oct 17 – Agent 1: Oct 25 - Agent 2: Oct 21 - Agent 2: Your phone is unreachable We assure you that Refund has been issued on refund has been issued 21st. 3-5 days for credit Oct 18 - Agent 1: Called at 7.40 pm. Still unreachable
Evolution of Customer Support Write to us IVR
Evolution of Customer Support Write to us IVR IVR Write to us Chat bot
Evolution of Customer Support IVR Write to us Chat bot
Refund Related Issues Refund Query If I cancel, how much refund will I get? Refund Delayed I have been waiting for too long Refund Status When can I expect to get the refund? Refund Special Claim • Flight was cancelled Refund Calculation Logic • Flight itinerary changed Explain how you determined the amount to be refunded
Deal with WhatsApp Lingo? Refund when? 😢 Hlo how to cancel Want 2 book flight for Kolkata early morning Father won’t travel with us. Pease cancel his ticket Cam I change name? maine flight ticket cancel ki thi, us may refund ishu kab tak me se kitna amount cut hoga? aayega kitna ayega
Issue Types: Fat head, Chunky middle and Long tail • Cancel my booking • Resend confirmation • Late check-in request • Cancel due to medical emergency • Require travel insurance certificate 20-30% of • Train delayed by more than 3 hours • Meal Included? issue volume • Claim travel insurance • Terminal details • Refund discrepancy • Extra charges at hotel • Buy insurance
Business Impact ~ 10K ~12 K Customers / Day Chats / Day 2 m Total Customers Served 7 % ~80 % Remaining volume that Bot Chats Handled by Bot could handle 80 % Bot CSAT / Agent CSAT
Architecture / NLU Model
Interactions Dialog Intent User Myra App APIs Manager Classifier Touch Type Speech Speak To Text Request(text input) Get Intent Invoke Action Respond
Interactions Dialog Intent User Myra App APIs Manager Classifier Touch Type Speech Speak To Text Request(text input) Get Intent Invoke Action Respond
Dialog Manager Turn 1
Dialog Manager Turn 2
Dialog Manager Turn 3 Your cancellation is successful! To know more about your refund, please choose one of the below options.
Frame Based State Tracker Frame Slot 1 Slot 2 Slot 3 confirm = Y Speech and Language Processing by Jurafsky & Martin Chapter 24: Dialog Systems and Chatbots
Frame Based State Tracker Frame Confirmation Booking ID Cancellation confirm success Show Do Show = Y Charges Cancel Result Speech and Language Processing by Jurafsky & Martin Chapter 24: Dialog Systems and Chatbots
Interactions Dialog Intent User Myra App APIs Manager Classifier Touch Type Speech Speak To Text Pl cancel booking Request(text input) MMT.Cancellation.FullCancellation Get Intent Invoke Action Respond
Intent Classifier Label Data Monitor Train Model Deploy Evaluate Model
Labelling Text Data Write to us Corpus Collect Label Define Analyse Data Check Frame Samples Samples Pl cancel booking Chat Labelled MMT.Cancellation.FullCancellation Corpus Tedious
Sequential Transfer Learning General Write Domain to us Labelled Corpus Fine-tune Fine-tune Evaluate Pretrain Language Intent Language Model Model Model Classifier • English Language • MMT Customer • Intent Specific • Micro F1-score increased Representations Support / Hinglish Representations from 0.80 to 0.89 Representations • Baseline: CNN + GloVe Build Intent Classifier with smaller amounts of labelled data
ULMFiT: Pretrained Language Model vs. Fine-tuned Language Model new > new york city is the only new delhi to mumbai flight i want > i want to be a real person I want to change my email to kitna > kitna ? kitna refund amt refund hoga Embedding LSTM LSTM LSTM Softmax Layer 1 2 3 Layer Universal Language Model Fine-tuning for Text Classification Jeremy Howard, Sebastian Ruder F E P F
ULMFiT: Pretrained Language Model vs. Fine-tuned Language Model Pretrained on Wikitext-103 Consisting of 28,595 preprocessed Wikipedia articles (103 million words) Fine-tuned on customer support corpus (~10 m words) Embedding LSTM LSTM LSTM Softmax Layer 1 2 3 Layer Universal Language Model Fine-tuning for Text Classification Jeremy Howard, Sebastian Ruder F E P F
ULMFiT: Pretrained Language Model vs. Fine-tuned Language Model Catastrophic Forgetting Discriminative Fine Tuning Different layers capture different • types of information Fine-tune each layer with different • learning rates Embedding LSTM LSTM LSTM Softmax Layer 1 2 3 Layer Universal Language Model Fine-tuning for Text Classification Jeremy Howard, Sebastian Ruder F E P F
ULMFiT: Pretrained Language Model vs. Fine-tuned Language Model Catastrophic Forgetting Slanted Triangular Learning Rates Model should adapt parameters • quickly to task specific features First, linearly increase the LR and • then linearly decay it Embedding LSTM LSTM LSTM Softmax Layer 1 2 3 Layer Universal Language Model Fine-tuning for Text Classification Jeremy Howard, Sebastian Ruder F E P F
Embedding: Vector Representation of cancel my flight 3 x 400 F E P F
ULMFiT: Fine-tuned Intent Classifier FullCancellation PartialCancellation F E P F
ULMFiT: Fine-tuned Intent Classifier FullCancellation PartialCancellation Gradual Unfreezing Gradually unfreeze starting from the last layer as this contains • information most specific to a domain F E P F Unfreeze the next lower layer. Repeat and fine tune all layers •
ULMFiT: Fine-tuned Intent Classifier FullCancellation PartialCancellation F E P F
Dealing with Intent Ambiguity F E P F
Analyzing Chats Frame 1 Slot 1 Slot 2 Slot 3 confirm = Y Frame 2 Slot 1 Slot 2 Slot 3 confirm = Y Frame 3 Slot 1 Slot 2 Slot 3 confirm Aggregate frame statistics = Y F E P F
Analyzing Chats CancellationPolicy Slot 1 Slot 2 Slot 3 confirm = Y FullCancellation Slot 1 Slot 2 Slot 3 confirm = Y RefundStatus Slot 1 Slot 2 Slot 3 confirm Aggregate frame statistics = Y F E P F
Analyzing Chats CancellationPolicy Slot 1 Slot 2 Slot 3 • Prob (frame j | frame i ) confirm = Y • Chat abandonment analysis • Agent transfer analysis • Last frame before transfer FullCancellation Slot 1 Slot 2 Slot 3 • Frame level analyses confirm • Input methods = Y • Touch • Type RefundStatus Slot 1 Slot 2 Slot 3 • Speak • Error analysis confirm = Y F E P F
Interactions Dialog Intent User Myra App APIs Manager Classifier Touch Type Speech Speak To Text Request(text input) Get Intent Invoke Action Respond Analyze & Improve KPIs
Issue Types: Fat head, Chunky middle and Long tail • Cancel my booking • Resend confirmation • Late check-in request • Cancel due to medical emergency • Require travel insurance certificate 20-30% of • Train delayed by more than 3 hours • Meal Included? issue volume • Claim travel insurance • Terminal details • Refund discrepancy • Extra charges at hotel • Buy insurance
Rate today’s session Session page on conference website O’Reilly Events App
Thank You Sanjay Mohan & Madhu Gopinathan www.makemytrip.com
Recommend
More recommend
