PREPARING FOR A UNIFIED IMC ARCHITECTURE BY 2020 STEVE WILKES - PowerPoint PPT Presentation

PREPARING FOR A UNIFIED IMC ARCHITECTURE BY 2020 STEVE WILKES 
 CO-FOUNDER & CTO OF STRIIM

EVERYTHING IS CONVERGING TOWARDS IN-MEMORY COMPUTING “ CONVERGENCE the merging of distinct technologies, industries, 
 or devices 
 into a unified whole ”

ENTERPRISE, CLOUD AND IOT ARE NOT ISLANDS Enterprise Cloud IoT

THEY ARE PART OF A CONNECTED ECO-SYSTEM Enterpris e Cloud IoT

PART OF A DIGITAL TRANSFORMATION THAT INCLUDES AI NLP Call Center/ Sentiment Predictive Analysis maintenance/ Retail Banking part management Enterprise Automating customer AI driven Machine engagement through smart bots adaptive AML Learning Cloud IoT CGM predictive Cybersecurity Analytics monitoring

EVERY INDUSTRY IS UNIFIED BY DIGITAL TRANSFORMATION Financial Services Healthcare Manufacturing Transportation/ 
 Retail Communication Logistics Public Sector IT Insurance

DATA GENERATION RATES ARE GROWING EXPONENTIALLY 95% Of Real-Time By 2025 This Will Data Will Be Leap to 160ZB Generated By IoT By 2025 25% Of All Data Will About 5% Of This Today We Generate be 
 Is Real-Time Data Around 16ZB Data Real-Time Annually Only A Small Percentage Of This Data Can Be Stored * Data Age 2025: The Evolution of Data to Life-Critical. An IDC White Paper, Sponsored by Seagate

IF YOU CAN’T STORE ALL DATA – WHAT CAN YOU DO? PROCESS AND ANALYZE DATA IN-MEMORY IN A STREAMING FASHION

NOT JUST IOT DATA IS MASSIVE – CYBERSECURITY FOR EXAMPLE How do you avoid losing or ignoring valuable data, while still storing only the minimum? 
 NETWORK 
 SECURITY 
 How do you correlate all events for immediate SERVERS & 
 SERVICES DEVICES DEVICES insights and proactive responses? How do you act promptly to better serve customers, protect reputation, and beat competitors?

REAL-TIME USE CASES CONVERGE ACROSS MANY INDUSTRIES Healthcare Financial Services Manufacturing - Proactive illness detection - Anti-money laundering - Quality management - Staff allocation optimization - Fraud prevention - Predictive maintenance - Point of care compliance - Risk management - Equipment monitoring - Eligibility verification - VIP customer service - Capacity optimization Communications Retail Transportation/Logistics - Fraud and theft detection - Connected car - Network health monitoring, - Real-time offers - Predictive maintenance - Predict network failures - Geo-targeted marketing - Asset tracking - Proactive service outreach - Dynamic pricing - Route optimization - Location-based advertising Public Sector IT Insurance - Crime detection and - Claim fraud detection - Cyber security prevention - Agent fraud detection - Replication validation - Cyber security - Risk-based policy pricing - API usage monitoring - Traffic management - Agency performance - SLA monitoring - Connected City - Usage-based insurance

ALL DATA ARRIVES IN STREAMS NOT BATCHES human s database streami machin es logs ng devic es events … stream processing has emerged as a major infrastructure requirement

STREAM PROCESSING REQUIRES A COMPLETE IMC PLATFORM VALUE EXTRACTED 
 IMMEDIATELY IN-MEMORY COMPUTING PLATFORM REAL-TIME 
 REAL-TIME 
 CONTINUOUS 
 CONTINUOUS 
 STREAM 
 STREAMING 
 DATA 
 INFORMATION 
 PROCESSIN ANALYTICS COLLECTION STORAGE G & ALERTING CONTEXT ADDED 
 WHILE PROCESSING

GARTNER TAXONOMY OF IN-MEMORY COMPUTING TECHNOLOGIES In-Memory Application Platforms In-Memory Analytics Stream Processing Other Application and Platforms Platforms Visual Data Discovery In-Memory Data Management Platforms High-Performance Message In-Memory In-Memory Infrastructure DBMSs Data Grids Memory-Intensive Computing Platform (DRAM, Flash, SSD, Multicore, InfiniBand, Clusters, Grid, Cloud) Source: Gartner (January 2017)

IN-MEMORY COMPUTING USED FOR HTAP & HIP In-Memory Application Platforms HTAP 
 In-Memory Analytics and Stream Processing Other Application Visual Data Discovery Platforms Platforms Hybrid Transactional 
 In-Memory Data Management Platforms Analytics High-Performance Message Infrastructure In-Memory In-Memory Processing DBMSs Data Grids In-Memory Application Platforms In-Memory Analytics and Stream Processing Other Application Visual Data Discovery Platforms Platforms HIP 
 Hybrid Integration 
 In-Memory Data Management Platforms Platform High-Performance Message Infrastructure In-Memory In-Memory DBMSs Data Grids

ALSO FOR STREAMING INTEGRATION AND ANALYTICS In-Memory Application Platforms In-Memory Analytics and Stream Processing Other Application Visual Data Discovery Platforms Platforms In-Memory Data Management Platforms High-Performance Message Infrastructure In-Memory In-Memory DBMSs Data Grids


 UNIFIED IMC ARCHITECTURE FOR STREAMING ANALYTICS Sources Targets Development Dashboards & Visualization 
 For In-Memory Analytics / Visual Discovery Streami ng Integrati on and Analytic Distributed Results Storage s Platform … Data Collection Stream Processing Data Delivery & Analytics CLUSTER Distributed In-Memory Data Grid Distributed High Speed Message Infrastructure

HOW DO YOU GET THERE? HYBRID OPEN SOURCE PROPRIETARY “OPEN CORE”


 BUILDING THIS FROM OPEN SOURCE Sources Targets Development Dashboards & Visualization Streami ng Integrati API Connectivity / Abstraction Layer / Web Server on and Analytic Distributed Results Storage s Platform From Glue-Code … Open Data Collection Processing & Data Delivery Source Analytics CLUSTER Clustering Scalability Reliability Security Management Distributed In-Memory Data Grid Distributed High Speed Message Infrastructure

OPEN SOURCE DEVELOPMENT PROCESS Build From Open Source Vendor Or Community Design Support For Each Component Maintain Identify Deprecated Build Test Deploy Applications Evaluate Upgraded Integrate Test

ADVANTAGES OF HYBRID “OPEN CORE” PLATFORMS HYBRID “OPEN CORE” Commodity meets OPEN SOURCE PROPRIETARY enterprise grade. Commodity Technology Combines rapid Business Logic Intensive Extensible Technology innovation & economies Unique Integration Critical Mass of commodity software of Niche Technologies Technology open source with security, unique IP, and last mile integration of proprietary

HYBRID “OPEN CORE” DEVELOPMENT PROCESS Build From Open Source Hybrid Support Design For Each Component Maintain Identify Deprecated Install Build Test Deploy Hybrid Applications Evaluate Upgraded Integrate Test

IMC NEEDS TO BE ENTERPRISE GRADE FOR MISSION CRITICAL APPS Scalability Reliability Enterprise Grade Security Integration

SCALABILITY "Scalability is a characteristic of Scalability Reliability a system that describes its Enterprise Grade capability to cope and perform Security Integration under an increased or expanding workload" Scalability in IMC: • Ingestion volume • Processing

RELIABILITY "Reliability is the ability of a Scalability Reliability system to consistently perform Enterprise Grade its intended or required function, Security Integration on demand without degradation or failure." Reliability in IMC: • Ingestion • Processing

SECURITY "Security is the mechanism by Scalability Reliability which a system is protected from Enterprise Grade data corruption, destruction, loss, Security Integration interception, or unauthorized access" Security in IMC: • Authentication • Authorization

INTEGRATION "Integration is the bringing Scalability Reliability together of component Enterprise Grade subsystems into one system and Security Integration ensuring that the subsystems function together." Integration in IMC: • Ingestion • Enrichment

EXAMPLE – STRIIM’S HYBRID ARCHITECTURE Sources Targets Drag and Drop UI Real-Time Streaming Dashboards 
 Kafka Kafka + Command Line Interface to Surface In-Memory Analytics HDFS HDFS Server Scalability, Distribution, Clustering & Failover TQL / JDBC / ODBC / REST / WS APIs Distributed Results Storage Role-Based Security & Encryption Elastic Reliability, Recovery & E1P Continuous 
 SQL-Based Continuous Management & Monitoring Data Processing Data … Collection And Analytics Delivery STRIIM 
 CLUSTER Databases (CDC) 
 Databases Files 
 Files Messaging Messaging Cloud Cloud Big Data Big Data Devices Distributed In-Memory Data Grid for Context Data JCache Distributed In-Memory Data Grid for Metadata / Control Hazelcast Flume HBase Distributed Persistent High Speed Message Infrastructure Kafka HBase Hive Distributed High Speed Message Infrastructure JMQ + Kryo