Instrumentation, Observability, and Monitoring of Machine Learning Models 1
About Me ● Google Engineer (2007-11) ● Cloudera’s Director of Data Science (2011-15) ● Slack’s Director of Data Engineering (2015-2017) ● Slack Engineer (now)
“”
“”
The Genesis of This Talk
Machine Learning In the Wild
Data Science Meets DevOps
Some History
Logs via the ELK Stack
Metrics with Prometheus
Prometheus Architecture
Traces
A Word About Cardinality
Make Good Decisions By Avoiding Bad Decisions
The ML Test Score
The Map Is Not The Territory
Monitor Model Decay
Build Lots of Models
Deploy Your Models Like They Are Code*
Stand On The Shoulders of Giants ● Ensembles ● Experiments ● Dark Tests ● Canary ● Sanity Checks
Tag All The Things
Circle of Competence
Garbage In...
Linking Online and Offline Metrics
Handling Cross-Language Feature Engineering
Know Your Dependencies
Monitoring For Critical Slices
Second-Order Thinking
On Razors
http://slack.com/careers 30
Recommend
More recommend
