1
play

1 Pivotal ConfidentialInternal Use Only Implementing a highly - PowerPoint PPT Presentation

Jan 06, 2023 •452 likes •686 views

1 1 Pivotal ConfidentialInternal Use Only Implementing a highly scalable Stock prediction system with Apache Geode, Spring XD and Spark MLib (incubating) Fred Melo William Markito @fredmelo_br @william_markito 2 About us Fred Melo

  1. 1 1 Pivotal Confidential–Internal Use Only

  2. Implementing a highly scalable Stock prediction system with Apache Geode, Spring XD and Spark MLib (incubating) Fred Melo William Markito @fredmelo_br @william_markito 2

  3. About us Fred Melo William Markito Technical Director for Data Enterprise Architect for GemFire fmelo@pivotal.io wmarkito@pivotal.io @fredmelo_br @william_markito

  4. A Simple Example Forecast Data Sources Look for patterns

  6. Applicability " Smart System"

  7. What do we want to build? Evaluates LIVE DATA “ According to historical Real-Time trends, there’s an 80% chance this stock prices might go down within the next few minutes" Trading Data Live data becomes historical over time Smart System Learns with HISTORICAL TRENDS Historical " How were the technical indicator readings when the latest price drops happened? "

  8. Data The Machine Learning Pipeline data flow Temperature Hot 1- Live data is ingested into the grid Apache Geode / GemFire Spring XD Live Data 4 - “Hot" data ages, becoming part of the 2 - Trained ML model compares historical dataset new data to historical patterns Machine 5 - Re-training 3 - Results are pushed Learning model triggered, ML Apache Hawq immediately to model updated. deployed applications Spring XD Cold

  9. Data The Machine Learning Pipeline data flow Temperature Simplified Model Hot 1- Live data is ingested into the grid Apache Geode / GemFire Spring XD Live Data 2 - Trained ML model compares new data to historical patterns Machine 3 - Results are pushed Learning model 5 - Re-training immediately to triggered, ML deployed applications model updated. Warm Spring XD

  10. 1 Transform Sink Split Real data /Stocks 2 Enrich Filter /TechIndicators 3 Simulator /Predictions Predict SpringXD Machine Indicators Learning Extensible Open-Source Dashboard Fault-Tolerant Horizontally Scalable Cloud-Native

  11. Too complex?? Eating it in small bites…

  12. GemFire SpringXD

  13. 1 Transform Sink Split Real data /Stocks 2 Enrich Filter /TechIndicators 3 Simulator /Predictions Predict SpringXD Machine Indicators Learning Extensible Open-Source Dashboard Fault-Tolerant Horizontally Scalable Cloud-Native

  14. Apache Geode Concepts • Cache • Configurable through XML, ,Java /Stocks • Region • Distributed j.u.Map on steroids /TechIndicators • Highly available, redundant • Member /Predictions • Locator, Server, Client • Callbacks • Listener, Writer, AsyncEventListener, Parallel/Serial

  15. Apache Geode HA and Fail-Tolerance

  16. 1 Transform Sink Split Real data /Stocks 2 Enrich Filter /TechIndicators 3 Simulator /Predictions Predict SpringXD Machine Indicators Learning Extensible Open-Source Dashboard Fault-Tolerant Horizontally Scalable Cloud-Native

  17. Streams 1 Transform Sink Split Pipelines SpringXD 2 Sources Enrich Filter Sinks 3 Predict Filters Taps

  18. 1 Transform Sink Split Real data /Stocks 2 Enrich Filter /TechIndicators 3 Simulator /Predictions Predict SpringXD Machine Indicators Learning Extensible Open-Source Dashboard Fault-Tolerant Horizontally Scalable Cloud-Native

  19. Label Features Machine Learning Model (e.g. Linear Regression) price(x) medium avg (x+1) medium avg (x) relative strength (x)

  20. Label Features Machine Learning Model (e.g. Linear Regression) price(x) medium avg (x+1) medium avg (x) relative strength (x)

  21. Demo Time Error

  22. Source code and detailed instructions available at: https://github.com/Pivotal-Open-Source-Hub/StockInference-Spark Follow us on Twitter! Fred Melo William Markito @fredmelo_br @william_markito 22

  23. 23 1 Pivotal Confidential–Internal Use Only

Recommend

More recommend