Building Efficient ML Pipelines and Responsible AI Solutions Adi - PowerPoint PPT Presentation

Building Efficient ML Pipelines and Responsible AI Solutions Adi Polak Microsoft @adipolak

Trust

• LET’S START FROM THE BEGINNING. What happens when we get raw data? @adipolak

@adipolak

ML Process / Life Cycle 1 Gather Data 2 Feature Extract, Clean and Normalize 3 Select algorithm Repeat! 4 Evaluate model 5 Data/Insights visualization @adipolak

@adipolak

But in real life: Accuracy < 0.5 ROC curve ☹ @adipolak

Aim for high Accuracy @adipolak

What can you do? Automate! @adipolak

HOW? Pipelines!

What are pipelines?

Big Data/ ML Pipelines Visualize Azure Machine Learning @adipolak

Demo Apache Spark ML Pipelines @adipolak

@adipolak Stepan Pushkarev, CTO, Hydrosphere.io

Big Data/ ML Pipelines Visualize Azure Machine Learning Azure Machine Learning @adipolak

High accuracy! But, at what cost? @adipolak

false positives @adipolak

Human centric Responsible AI @adipolak

Pixabay

Our updated goals: Lawful Ethical Robust @adipolak

ML is a black box ML algorithm Training: Model Data Data Model Testing/Prediction: Prediction @adipolak

Big Data/ ML Pipelines Visualize Azure Machine Learning Azure Machine Learning @adipolak

Big Data/ ML Pipelines Visualize Azure Check and transform the Machine Learning data s r e Balance the data n i a l p x Visualize the model E @adipolak

How ONLINE FREE INVESTED 1B$ IN Microsoft HIGH QUALITY OPEN AI support COURSES Responsible AI 115M$ GRANT FOR OPEN SOURCE AI FOR GOOD @adipolak aka.ms/free-responsible-ai-cour aka.ms/ml-interpretability-to se ol

@adipolak aka.ms/ml-interpretability-to ol

aka.ms/ml-interpretability-to ol

Azure Cognitive Services aka.ms/AA6kex s @adipolak

Tools Spark Spark ML Streaming MLflow Spark SQL @adipolak

But in real life: Accuracy < 0.5 ROC curve ☹

Demo Apache Spark ML Pipelines with Cognitive Services @adipolak

Demo @adipolak

You are only Good as your Data is Use explainers Understand your data @adipolak

Learn more ! Thank you ! aka.ms/free-responsible-ai-course aka.ms/twitter_sentiment_analysis aka.ms/ml-interpretability-tool aka.ms/ai-for-good-grant @adipolak

What is Machine learning • Lifecycle: • Gather data • Data preparation – clean it • Data wrangling • Data analysis • Feature extraction • Train model • Test model • Deployment