biophotonics
play

BIOPHOTONICS With PredictionIO, Spark and Deep Learning Prajod - PowerPoint PPT Presentation

Jun 05, 2023 •769 likes •1.3k views

ApacheCon Big Data North America May 2017, Miami, USA BIOPHOTONICS With PredictionIO, Spark and Deep Learning Prajod Vettiyattil, Architect, Wipro @prajods https://in.linkedin.com/in/prajod 2 ABOUT ME Architect at Wipro Big Data

  1. ApacheCon Big Data North America May 2017, Miami, USA BIOPHOTONICS With PredictionIO, Spark and Deep Learning Prajod Vettiyattil, Architect, Wipro @prajods https://in.linkedin.com/in/prajod

  2. 2 ABOUT ME • Architect at Wipro • Big Data division of Open Source Solutions team • Machine Learning • Video Analytics • Platform design and implementation • Domain solutions • Spark, Java, Python, DL4J, Tensorflow Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  3. 3 AGENDA • Bio photonics • Applications • PredictionIO • Apache Spark • DeepLearning4J and Tensorflow • Cell detection process • Deep learning and CNN • Solution Architecture Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  4. Biophotonics using Apache PredictionIO, Spark and Deep Learning 4 #apacheconbigdata @prajods SESSION OVERVIEW In 4 slides

  5. 5 APPLICATIONS • Self driving cars • Robots • Drones • Industrial automation • Physical security • Medical labs • Wherever images or videos are used Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  6. 6 SESSION OVERVIEW • Need in the healthcare domain • Speed up and automate, cell detections, counting and analysis • Diagnosis • Medical research • Solution • Train a Deep Learning Model using digital images of living cells • Recognize test images with high accuracy • Technology used • Training process Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  7. 7 CLASSIFICATION NEED Input from the microscope Expected output Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  8. 8 CLASSIFIED OUTPUT Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  9. Biophotonics using Apache PredictionIO, Spark and Deep Learning 9 #apacheconbigdata @prajods BACKGROUND How its done

  10. 10 INTRODUCTION • Photonics: study and harness light • The World of Small Things • Microscopic life • High end microscopes • Data set scarcity • Accessibility nigms.nih.gov Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  11. 11 LIVE CELL IMAGING Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  12. 12 CONFOCAL MICROSCOPE • Very high resolution • Spatial features Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  13. 13 IMAGE COMPARISON meyerinst.com Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  14. 14 ELECTRON MICROSCOPE Ref: emc.sc.edu Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  15. 15 What to do with all these images of micro stuff ? Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  16. 16 Spend hours peering through the lens ? Ref: wisegeek.org Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  17. 17 • Even then • How many cells can one count in a minute ? • How accurate is our ability to visually differentiate between bacterium A vs bacterium B ? • How many patient blood samples can one analyze in an hour ? • Can a doc detect all abnormalities with his endoscope ? • How accurate is human visual diagnosis ? Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  18. 18 AUTOMATED ANALYSIS OF CELLS • Detection of cells • Count cells • Distinguish cell A vs cell B • Detect physical abnormalities • Cell lifecycle analysis Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  19. Biophotonics using Apache PredictionIO, Spark and Deep Learning 19 #apacheconbigdata @prajods TECHNOLOGY

  20. 20 PREDICTION IO • Simplifies Machine Learning projects • Data storage • Training • Evaluate models • Deploy models • Serving predictions Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  21. 21 PREDICTION IO • DASE architecture • Data • Algorithm • Serving • Evaluation Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  22. 22 PREDICTION IO • Readymade ML templates • Classification • Regression • Recommendation • NLP • Clustering • Similarity Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  23. 23 PREDICTIONIO: LOGICAL VIEW Storage Event Server Other Other Other Training Engine Client components components components application Serving Engine Evaluator PredictionIO Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  24. 24 PREDICTIONIO: PRODUCT VIEW Storage Event Server (Hbase/Postgres/MySQL) (Spray+Storage) Other Other Other Training Client components components components application Engine(Spark) Serving Engine (Spray+Spark) Evaluator PredictionIO Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  25. 25 APACHE SPARK • Fast in memory data processing • Real time and batch modes • Complements Hadoop • Replaces Hadoop MR • Adds • In memory processing • Stream processing • Fast for interactive queries • YARN or Mesos for clustering • Java, Scala, Python, R Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  26. 26 SPARK: LOGICAL VIEW Spark Spark SQL SparkML GraphX Streaming Apache Spark Core Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  27. 27 SPARK: DEPLOYMENT VIEW Cache Task Task Task Executor Executor Spark Driver Worker Node Spark’s Cluster Cache Task Task Manager Task Executor Executor Executor Master Node Worker Node Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  28. 28 DEEPLEARNING4J (DL4J) • Deep learning library • Open source • Apache 2.0 license • Java based • Distributed execution • Runs on Spark and Hadoop Smart Manufacturing with Apache Spark and Deep Learning #apacheconbigdata @prajods

  29. 29 TENSORFLOW • Deep Learning framework • from the Google Brain Team • Python and C++ SDKs • Dataflow graph based processing • Tensors and Operations • Numerical operations • Lazy evaluation • Distributed and parallel • Training and inference • Good documentation • Useful examples Ref: tensorflow.org Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  30. 30 TENSORFLOW • CPU, GPU • Mobile: IOS and Android • Core API in C • Compiled models • Visualization using TensorBoard • Tensorflow Serving Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  31. Biophotonics using Apache PredictionIO, Spark and Deep Learning 31 #apacheconbigdata @prajods WHAT DOES IT INVOLVE ?

  32. 32 THE CELL DETECTION PROCESS • Data gathering • Data preparation • Data extraction • Model training • Evaluation Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  33. 33 DATA GATHERING • “Google” it ? • Cell image data sets are not common • Very few youtube videos • Get the data set from the labs • Caveat: Competitive information davidbarlowarchive.com Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  34. 34 DATA EXTRACTION • Extract your own data sets from videos • Different angles, lighting, perspective • Multiple cells • Image processing techniques • Edge detection • Segmentation • Back ground subtraction • Otsu • Watershed Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

  35. 35 MODEL TRAINING • Custom models • Build your own • High difficulty in hyper parameter tuning • Very high training effort • Small sizes • Poor accuracy • Transfer learning • Reuse an existing image detection model • Tensorflow’s inception • Replace its final layer/s • Very little hyper parameter tuning • Involves lower training time Biophotonics using Apache PredictionIO, Spark and Deep Learning #apacheconbigdata @prajods

More recommend