leveraging aws and machine learning to power search at
play

Leveraging AWS and Machine Learning to Power Search at Zocdoc - PowerPoint PPT Presentation

Nov 25, 2022 •220 likes •582 views

Leveraging AWS and Machine Learning to Power Search at Zocdoc Pedro Rubio Head of Search Engineering Brian dAlessandro Head of Data Science This document and its contents are proprietary and confidential of Zocdoc, Inc. and may not be

  1. Leveraging AWS and Machine Learning to Power Search at Zocdoc Pedro Rubio Head of Search Engineering Brian d’Alessandro Head of Data Science This document and its contents are proprietary and confidential of Zocdoc, Inc. and may not be reproduced or shared, in whole or in part, without the express written authorization of Zocdoc, Inc.

  2. Agenda - How we’re built - People and Architecture - How we’re built - the Data - Questions

  3. Problem Statements: 1. Patients need to find and book with a doctor, and, 2. Patients don’t often know what kind of doctor they need. 3

  4. 4

  5. How we’re built And solving “what the patient means”

  6. Core Optimization Problems for ZD Search • Cross team collaboration enabling maximum iteration speed • Deliver recommendations < 200 ms • Patient satisfaction (And our architecture plays a big role here!) 6

  7. The Search Team Product Engineering Data Science Design 7

  8. Zocdoc Tech Stack NodeJS, ES6, Babel, React - AWS - Cloudformation, Docker, ECR, - EC2, ELB Kinesis / Firehose - S3 - reporting to data-lake - Monitoring with Datadog - Routes with Express -

  9. Our Legacy Search

  10. Free Text (Patient Powered) Search

  11. Types of Intent Name of doctor Medical procedure Specialty Symptom 11

  12. Intent Parsing Architecture Design Machine Learning 12

  13. Doctor Name Retrieval Specialty Retrieval Browser Visit Reason Retrieval Phase I - Auto-Suggest NLP Search Semantic Retrieval Service Pipeline Semantic Corpus Building Service Service Handler Auto-Suggest Ranking Results Ranking Models Logging Phase II - Backend Search

  14. Solving for the Long Tail The structured queries comprise a reasonable percent of traffic, but are a minority of total search terms we service. We use Natural Language Processing (NLP) algos to map unstructured terms into our structured search set. Specialties = O(10^2), Procedures = O(10^3), Names = O(10^6), Other = O(10^7) 14

  15. Different Representations of Same Concept Variations Concept Medical Term Interpretation • Heart beats too • Irregular • Atrial fast heartbeat Fibrillation • Heart flutters • Heart • Pulse rate too high palpitations • Irregular pulse • Heart out of rhythm • Irregular heartbeat • Heart palpitations 15

  16. ZocDoc Semantic Service f(“presentation anxiety”) = {[{Specialty =“Psychologist”, Relevance = 0.8}, …,{Specialty =“Psychiatrist”, Relevance = 0.7}]} 16

  17. Early Results (And Why You Need to Always Experiment)

  18. Searches that Lead to “Nephrology” Many patients don’t know what a Nephrologist is. They don’t need to know to find one now. 18

  19. How We’re Built - The Data

  20. Data - Indexing so we can Search

  21. Lesson Learned with Indexing Data Legacy Layer AWS Elastic.co Monolith Live Feed Process S3 ƛ Cache Lambda’s act as a mini ETL layer getting the documents ready for our retrieval stage. - Complex “stateless” ETL process that - Lambda memory max 1500mb transforms this data into the data that - Our data much larger we need in Elasticsearch - Manage state in S3 and - Load piecemeal into Elasticsearch Elasticsearch - At the very end, swap alias to use newly uploaded indexes

  22. New ETL - Spark - Spark - ETL - Get over 1500mb limit - Get over 5 minute runtime limit More complex Processing - Easily add more data-sets 1 - Currently in Databricks - Plan to migrate to EMR (Elastic Map Reduce) 2 3 Mapping from Mapping from 1 -> 2 1 -> 3 Joined Data Set Business Logic Application

  23. Data - Event Data So we can Learn

  24. The Marketplace Goal: Make it as easy as possible to match the user to the right doctor. Considerations: • How to weight distance vs. availability vs. experience vs. reviews? • Does Dr. take this type of patient? • Are we meeting regulatory requirements? 24

  25. Organizational Optimization Optimize: algo iteration speed Subject to: • Org too small to justify full time data scientists within search • Throwing models over the wall to be implemented doesn’t work 25

  26. Agile Machine Learning Production Filtered Model API Results ZocDoc Query Model dB Prod Service Transformations + Scored Ranked Model Scoring Results (Search) Results Research, Analysis, Logs Model Development (S3/Redshift) (Spark/Redshift) Offline Engineering Owned DS Owned 26

  27. Aqueduct: Filling the Data Lake Some Data Lake principles: • Allow producers to easily push data • Allow data format changes • Smart ETL to make consumption very easy 27

  28. Cistern: Making Datalake Drinkable • “Raw” data lake good for exploratory research (we use Spark) • “Clean” data lake better for analytics and quick exploration 28

  29. Data - Insights

  30. 30

  31. We’ve got our fingers on Searches for Therapy/Therapist on Zocdoc the pulse of public health trends. 31

  32. How Much is that Smile Worth? Click Conversion by Search Rank and DrIsSmiling We’re exploring AWS Rekognition to research what drives user interest in Dr. profiles. 32

  34. 34

  35. Thank you and Questions!

Recommend

Example 1 : Selecting a present for somebody You enter a large shop with no idea what

Example 1 : Selecting a present for somebody You enter a large shop with no idea what

Evolutionary Computing (Genetic Algorithms) Heralded as an approach to machine learning Learning new solutions/behaviours by evolving old ones Also an approach to search Search in a potentially infinite search space No

321 views • 6 slides
The Duck Test: Leveraging Machine Learning to Remediate Fraud in Huge Datasets Matthew Harper

The Duck Test: Leveraging Machine Learning to Remediate Fraud in Huge Datasets Matthew Harper

1 The Duck Test: Leveraging Machine Learning to Remediate Fraud in Huge Datasets Matthew Harper Director Cyber Crime Prevention 2 Who is Aflac? Supplemental insurance Significant presence in Japan (2/3 of corporate revenue) My

324 views • 18 slides
Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t

Akinori Tanaka (RIKEN AIP/iTHEMS) Machine learning techniques to probe theoretical physics Intro In inSPIRE, search find t machine learning OR deep learning and date 20xx-&gt;20xx+1 year number of results Intro Mainly experimental, a few

908 views • 70 slides
Leveraging Machine Learning to Improve Unwanted Resource Filtering Sruti Bhagavatula Christopher

Leveraging Machine Learning to Improve Unwanted Resource Filtering Sruti Bhagavatula Christopher

Leveraging Machine Learning to Improve Unwanted Resource Filtering Sruti Bhagavatula Christopher Dunn Chris Kanich Minaxi Gupta Brian Ziebart 1 Introduction 2 Introduction 3 Typical Advertisement Typical DOM

529 views • 23 slides
Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1

Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1

10-418 / 10-618 Machine Learning for Structured Data Machine Learning Department School of Computer Science Carnegie Mellon University Learning to Search + Recurrent Neural Networks Matt Gormley Lecture 4 Sep. 9, 2019 1 Reminders

599 views • 58 slides
Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor

Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor

Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be

770 views • 57 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
Role of Pricing in Leveraging Market Power Role of Pricing in Leveraging Market Power Tom Hird

Role of Pricing in Leveraging Market Power Role of Pricing in Leveraging Market Power Tom Hird

Role of Pricing in Leveraging Market Power Role of Pricing in Leveraging Market Power Tom Hird (Ph.D.) Background Background Monopolists often operate, or want to operate, in markets that are downstream to their monopoly infrastructure

674 views • 16 slides
10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10d Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

1.05k views • 29 slides
10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10b Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

866 views • 50 slides
10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10a Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

935 views • 54 slides
10c Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10c Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A

10c Machine Learning: Symbol-based 10.0 Introduction 10.5 Knowledge and Learning 10.1 A Framework for 10.6 Unsupervised Learning Symbol-based Learning 10.7 Reinforcement Learning 10.2 Version Space Search 10.8 Epilogue and 10.3 The

605 views • 35 slides
Retrieval as Interaction African Summer School on Machine Learning for Data Mining and Search

Retrieval as Interaction African Summer School on Machine Learning for Data Mining and Search

Retrieval as Interaction African Summer School on Machine Learning for Data Mining and Search Maarten de Rijke January 14, 2019 University of Amsterdam derijke@uva.nl Based on joint work with Abhinav Khaitan, Ana Lucic, Anne Schuth, Boris

1.16k views • 50 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Signal Processing and Machine Learning for Power Quality Disturbance Detection and Classification

Signal Processing and Machine Learning for Power Quality Disturbance Detection and Classification

Signal Processing and Machine Learning for Power Quality Disturbance Detection and Classification Roberto Togneri (Signal Processing and Recognition Lab) Power Quality (PQ) disturbances are broadly classified into three categories: magnitude

407 views • 23 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
Machine Learning for Performance and Power Modeling/Prediction Lizy K. John University of Texas

Machine Learning for Performance and Power Modeling/Prediction Lizy K. John University of Texas

PACT 2010 Machine Learning for Performance and Power Modeling/Prediction Lizy K. John University of Texas at Austin Simulation Challenges Simulation Based Performance Models eg: SimOS, SIMICS, GEM5, SimpleScalar Power modeling eg:

778 views • 39 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
321 [No trash found] Clean up Head for Search [Seen trash] [Recharged] trash Get power [No

321 [No trash found] Clean up Head for Search [Seen trash] [Recharged] trash Get power [No

5.3 State Machines 321 [No trash found] Clean up Head for Search [Seen trash] [Recharged] trash Get power [No power] [Got item] [Trash disposed] H* Head for compactor Figure 5.17 A hierarchical state machine with a cross-hierarchy

534 views • 10 slides

More recommend