harish@datatron.com jerry@datatron.com Lessons Learned Deploying and Monitoring AI Models in Production at Major Tech Companies
Who are we? Harish Doddi CEO Jerry Xu CTO datatron 2
Today’s Enterprise AI life cycle Development: Optimization Production Discovery “The Battleground” “The Battleground” “The Playground” datatron 3
Lesson 1 You either NEVER deploy a model, or you have to do it over and over again datatron 4
Are your models decaying? Deploy over and Deploy and Done over… Model decays over time Model replenishes Performance Performance Time Time Model performance Model performance Result Result decreases consistent datatron 5
ML model cycle is a continuously optimizing process Model Building Concept drift New concept comes up … Model Model Monitoring Deployment Model Model Management Testing datatron 6
Connecting Machine Learning to Software world Before Now Future Software deployment Machine Learning models will deploy every day very frequent Software deployment and fast BUT once a 1 or 2 years Machine Learning models deploy very slow datatron 7
Lesson 2 Models may go wrong, you need to monitor them datatron 8
South Park and Alexa datatron 9
Monitoring Learning: Post mortem is the only option Without Model Monitoring The problem The team detects the problem and occurs decides what to do With Model Monitoring The problem The team decides occurs what to do Notify asap datatron 10
Monitoring for Machine Learning Models Model Performance monitoring • Confusion Matrix • Gain and Lift charts • Kolomogorov Smirnov chart • Area Under the ROC curve • Gini Coefficient • Concordant – Discordant ratio • Root Mean Squared Error (RMSE) • etc Model Timeout monitoring Infrastructure monitoring Organization KPI monitoring Deployment monitoring datatron 11
Lesson 3 Your real work starts AFTER you deploy the model to production datatron 12
Enterprise AI Life Cycle Exploration Training Deploy datatron 13
Enterprise AI Life Cycle After Deployment Model Anomaly Deploy A/B Testing SLA Selection detection Blue Green Split traffic Monitor performance Model routing Feature distribution Deployment Shadowing Fall back strategy Challenger Model result Rollback Alerting KPI based selection Canary datatron 14
Lesson 4 Data science is scarce resource, you need to make sure you organize well datatron 15
Deployment Learning: Rise of new engineering role Machine Learning Deep Learning Engineer Engineer There is a hyper-competitive WAR FOR TALENT that is projected to get much worse datatron 16
Teams face cross- End User functional Production Machine Data Science DevOps Engineering Model Learning inefficiencies BEST CASE SCENARIO: with a world-class team, 1 model deployed • Teams operate in silos, don’t speak the same language per quarter • Errors due to lack of communication • Engineering has to write stand-alone scripts datatron 17
Hidden Technical Debt in Machine Learning Systems Google Paper Hidden Technical Debt in Machine Learning Systems Boundary erosion Machine learning offers a fantastically powerful Entanglement toolkit for building complex systems quickly. … it Hidden feedback loops is remarkably easy to incur massive ongoing Undeclared consumers Data dependencies maintenance costs at the system level when Changes in the external world applying machine learning. System-level anti-patterns datatron 18
Lesson 5 Be prepared, your number of models will increase datatron 19
Deployment Learning: 1 model vs Multiple models datatron 20
Cost per model increases significantly if no automation As the number of models increases, the cost also increases Cost per Cost per model model # of # of models models datatron 21
Lesson 6 Senior people are needed AFTER deploying to production datatron 22
Software Development vs ML Model Development Senior People Requirements Testing Evolution Design Implementation Senior People Monitoring Data Training / Deploy to Requirements and Preparation Testing Production Optimization datatron 23
Lesson 7 Don’t be married to a single framework datatron 24
Build/Bring Your Own Models, Frameworks, Languages datatron 25
Thank you! Innovators Pavilion Booth P4 harish@datatron.com
Recommend
More recommend