Big Data & BI – The Most Powerful Combo Asha Subramanian HCL Technologies Ltd. Using and Acknowledging Sources Presenter: David Stodder, TDWI – Big Data and BI : How they come together
AGENDA � Where BI is going � Big Data Trends and What’s different about Big Data � Keys to Complimentary BI and Analytics with Big Data � Concluding Thoughts 2
BI’s Mission – Responding to Business Needs � Speed � Reduce, if not eliminate delay in servicing customers, responding to market events and increasing efficiency � Agility � Control and coordinate access across internal and external business networks to improve versatility and reaction � Intelligence � Use information to be predictive and proactive, learn from multiple data sources to continuously improve decisions � Effectiveness � Manage costs and increase productivity 3
Where’s BI been and where it is going PAST & PRESENT PRESENT & FUTURE • Small, departmental user communities • Enterprise Deployments; Diverse dependent on IT and “Power Users” users requiring self service functionality • ETL Processes prepare data for • ELT, in memory Analytics, “Big Data” specific use ( eg. reports ) access to raw sources, detailed data, more sources • Focused on structured data sources for reports and Ad-hoc query analysis for reports and Ad-hoc query analysis • Needed – Portfolio of search, query, • Needed – Portfolio of search, query, Richer metadata for semi structured • Historical “Rear View Window” on the data • Past, Present and Future data views and analysis 4
Changing BI Imperatives Conquer Data Latency Enable Self Service BI and Analytics - Faster Data Delivery - Business Users want from canned reports to be in control and to real time not IT dashboards dashboards - Users need structured - From batch, historical data and unstructured reports to alerts and content exception reports - Users want to discover - Operational users data in guidance need automated decisions IMPERATIVE # 1 IMPERATIVE # 2 5
Changing BI Imperatives Operational Intelligence Enhancing Visibility into Data - Ability to understand, analyse and act on - Dashboards : Visual, continuous stream of role based views of data (spatial, sensors, actionable data social media, mobile social media, mobile device data etc..) - Specialised visualisations for data types, models - Visibility: Integrated and analytics views of data across multiple event streams - Mobile BI: Synchronised and data sources to visualised for people on spot correlations and the go. Location patterns intelligence IMPERATIVE # 3 IMPERATIVE # 4 6
Expanding Universe of Data Sources - 204.243.130.5 - - [26/Feb/2001:15:34:52 -0600] "GET / HTTP/1.0" 200 8437 "http://metacrawler.com/crawler?general =dimensional+modeling" "Mozilla/4.5 [en] (Win98; I)" [en] (Win98; I)" Business Application Data Machine Generated Data Human Generated Data Highly Structured Arbitrarily Structured 7
BIG DATA is … � BIG NEWS !!!!!! � BIG ( Terabytes of Data, Petabytes soon, what next ….) -> VOLUME � FAST (continuous, regular intervals, bursts…) -> VELOCITY � ANY FORM ( structured, semi structured, unstructured ) -> VARIETY � IS NOT UNIFORM (does not conform to predictable structures..) -> VARIABILITY � IS NOT UNIFORM (does not conform to predictable structures..) -> VARIABILITY � IS EVERYWHERE � ………………………………………IS FUELLING THE NEW AGE OF ANALYTICS 8
BIG DATA – Why is it important ? � ONE DEFINITION – It lies beyond the capabilities of the current BI and Data Warehousing � BEYOND RELATIONAL - � BEYOND RELATIONAL - Flow of semi-structured or unstructured content � BEYOND STRUCTURE – Data complexity defies current BI metadata and structure � BEYOND THE WAREHOUSE – Demand for Hadoop (HDFS), MapReduce, NoSQL 9
Big Data Sources – Variety and Velocity 10
BIG DATA – Volume : TB today, PB soon � Users conduct analytics with ever larger data sets � A third of surveyed organisations have crossed the 10 TB barrier � Soon, we will measure Big Data in Petabytes and not terabytes 11
Big Data and Transformation to BI � Storage – Largely relational or � Storage – NoSQL, Hierarchical, Key Columnar Value Stores, Document based stores, Column oriented databases � Limitations on scalability, requires � costly high end storage to scale to Has the capability to scale on larger volumes of data commodity hardware � Standard BI/DW : Schema created, � Big Data – Don’t transform it, just put it then data loaded and transformed as then data loaded and transformed as into a file – raw (egs, Indexed, Key- into a file – raw (egs, Indexed, Key- per internal data structure Value pairs..) � Data extracted as per the analytics � Late Binding – Let Big Data Analytics requirement determine extract required at read time, find the structure in the data 12
BIG Data Technologies Convergence of many innovations – Technology CoE Stream Scope Big data • Map-reduce implemented by Hadoop, SQL-MR alternative from AsterData, eclipse based MR IDE programming from Karmasphere • Event Stream Processing (ESP, implemented by IBM Streams, Oracle CEP, Esper and many others) • Complex Event Processing (CEP, implemented by TIBCO BE, IBM Websphere Operational Decision Management) • Tools that make the use of these new programming techniques more easily accessible to business users, e.g. IBM BigSheets Big data • NOSQL databases for scale-out approach to managing structured and/or unstructured data application data Eg. Cassandra, CouchDB .. stores stores • In-memory databases that allow eXtreme Transaction Processing (XTP) and high-performance • In-memory databases that allow eXtreme Transaction Processing (XTP) and high-performance through distributed caching. Oracle Coherence, VMWare GemFire .. Big data OLAP • Appliances such as Exadata and Teradata Platforms • Shared-nothing analytical databases like EMC Greenplum • Columnar databases like HP Vertica • In-memory appliance like SAP Hana • In-memory self-service analytics and data visualization platforms like Spotfire, Qlikview, Tableau, Splunk Analytics • SAS e-miner, SPSS, Revolution analytics, Spotfire analytics and the like (Data science) • Customer experience analytics like ClickFox • Text mining tools like SAS text miner and OpenCalais • Social intelligence platforms like Radian6 and Visible Pan-Enterprise • Autonomy, FAST ESP, Clarabridge etc. Search 13
What is a Big Data appliance? - Integrates key components of a big data platform into a single product - No risks of a custom built solution - Comes with inbuilt connectors - Storage, Processing, Analytics and Visualisation platforms all bundled into one product 14
Oracle Big Data Implementation 15
IBM Big Data Implementation Data warehouse Data Integration Analytics Platform Applicance Platform - Infosphere Streams - IBM Netezza - Infosphere Information - IBM Infosphere - Infosphere Warehouse Server BigInsights Storage Organise Analyse and Decide 16
Operational Business Intelligence Machine Data -Unstructured Data -Tremendous Source of Business Value -Cannot be handled by BI -Cannot be handled by BI -Under Leveraged -Needs New Approach Structured Data -Business Txn data -Well understood -Handled by traditional BI -Slow growth Machine Data Comprises of data from RFID, GPS, Sensors, Web Servers, Messaging, Clickstreams, Mobile devices, Databases, Telematics, Servers etc.. 17
Operational Intelligence Tools - Examples Storage Real Time Analyse Collection & Indexing Visualise Explore, Analyse And Visualize Big Data And data from Analytic In memory & Databases and Cubes In Database Analysis Use R Statistical Analysis Software 18
Business Visibility from Big Data and traditional BI - example Unstructured Data from Master Data from Location Customer Interaction Information Traditional Data Warehouse Customer Interacts with Location information Correlated with relevant service Online or from any based on where the information from the DB device customer interacted with the service REAL TIME BUSINESS INSIGHTS •What products are popular in what •What are interaction paths by regions devices •Which products are customers •How can we improve customer leaving in cart experience 19
Big Data & BI – Most Powerful Combo Big Data for BI : Improving Operations � Enriching the data: Going beyond the limits of just structured relational data to show a complete view, including data relationships across sources � Using Big Data technologies to gain a near or true real time view of data flowing into and through the organisation � Not waiting for the data warehouse: Implementing models and sensors to spot exceptions � Not waiting for the data warehouse: Implementing models and sensors to spot exceptions or patterns in large data as they are happening - not as historical data � Integrating search for accessing semi structured or unstructured sources : finding the structure in the source rather than accessing only data that conforms to predetermined structure 20
Recommend
More recommend
