of big data
play

of Big Data 10/01/2018 25 Storage of Big Data Data is growing - PowerPoint PPT Presentation

Oct 09, 2022 •294 likes •460 views

Components of Big Data 10/01/2018 25 Storage of Big Data Data is growing faster than Moores Law Too much data to fit on a single machine Partitioning Replication Fault-tolerance 10/01/2018 26 Hadoop Distributed File System (HDFS)

  1. Components of Big Data 10/01/2018 25

  2. Storage of Big Data Data is growing faster than Moore’s Law Too much data to fit on a single machine Partitioning Replication Fault-tolerance 10/01/2018 26

  3. Hadoop Distributed File System (HDFS) The most widely used distributed file system Fixed-sized partitioning 3-way replication Write-once read-many … 128MB 128MB 128MB 128MB 128MB 128MB … 10/01/2018 27

  4. Indexing Data-aware organization Global Index partitions the records into blocks Local Indexes organize the records in a partition Challenges: Global index Big volume HDFS limitation New programming paradigms Local indexes Ad-hoc indexes 10/01/2018 28

  5. Fault Tolerance Replication Redundancy Multiple masters 10/01/2018 29

  6. Streaming …1000100010101011101110101010110111010111011101110100… Processing window Sub-second latency for queries One scan over the data (Partial) preprocessing Continuous queries Eviction strategies In-memory indexes 10/01/2018 30

  7. Task Execution MapReduce … M 1 M 2 M m Map-Shuffle- Reduce Resiliency through R 1 R 2 R n materialization Resilient Distributed Datasets (RDD) Directed-Acyclic-Graph (DAG) In-memory processing Resiliency through lineages Hyracks Stragglers Load balance 10/01/2018 31

  8. Query Optimization Finding the most efficient query plan e.g., grouped aggregation Agg Partition Merge Agg Vs Agg Partition Merge Agg Agg Partition Cost model (CPU – Disk – Network) 10/01/2018 32

  9. Provenance Debugging in distributed systems is painful We need to keep track of transformations on each record 10/01/2018 33

  10. Big Graphs Motivated by social networks Billions of nodes and trillions of edges Tens of thousands of insertions per second Complex queries with graph traversals 10/01/2018 34

  11. Hadoop Ecosystem Administration Pig MapReduce Query Engine Yet Another Resource Negotiator (YARN) Hadoop Distributed File System (HDFS) 10/01/2018 35

  12. Spark Ecosystem Spark SQL Spark Data Frames MLlib GraphX SparkR Streaming Resilient Distributed Dataset (RDD) a.k.a Spark Core Yet Another Kubernetes Resource Negotiator (YARN) Hadoop Distributed File System (HDFS) 10/01/2018 36

  13. AsterixQL HiveQL PigLatin MapReduce Pregel Jobs Jobs Other AsteixDB HiveSterix Hyracks compilers jobs Algebricks Hadoop MapReduce Pregelix Algebra Layer Compatibility Hyracks Data-parallel Platform 10/01/2018 37

  14. Impala Query Parser Query Planner Query Executor Yet Another Resource Negotiator (YARN) Hadoop Distributed File System (HDFS) 10/01/2018 38

  15. SpatialHadoop Pig Latin + Pigeon Spatial Visualization MapReduce Processing + Spatial Query Processing Yet Another Resource Negotiator (YARN) Hadoop Distributed File System (HDFS) + Spatial Indexing 10/01/2018 39

  16. Reading Material “The Age of Analytics in a Data - driven World” [Executive Summary] by McKinsey & Company 10/01/2018 40

Recommend

Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data mining is the use of efficient techniques for the analysis of very large collections of data and the extraction of useful and possibly unexpected

2.4k views • 94 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 68 / 91 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4 2

579 views • 8 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 28 / 89 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4

455 views • 23 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 51 / 89 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4

344 views • 17 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 68 / 91 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4 2

729 views • 24 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and repairing account for about 60% of the work of data scientists." Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides
Data stories with The Pudding So what is data storytelling? Data academia & data science

Data stories with The Pudding So what is data storytelling? Data academia & data science

Data stories with The Pudding So what is data storytelling? Data academia & data science dense complex static/non-interactive often not accompanied by an easy-to-follow narrative Data art beautiful abstract

647 views • 5 slides
Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by

Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by

Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by some geophysical device Most surveys are comprised of linear traverses or transects Typically constant data spacing Perpendicular to

432 views • 12 slides
Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges:

Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges:

Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges: data Data type: numerical sensor readings. Rich and massive data, spatially distributed and correlated. Data dynamics: data

714 views • 48 slides
Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data

Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data

DATA Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data Obtaining data Further study THE ROLE OF DATA Data refers to observed facts there are 82 persons in this train the weight of

683 views • 22 slides
DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type

DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type

DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type system sets the stage for the capabilities of the language Understanding data types empowers you as a data scientist DataCamp Data Types for Data

527 views • 23 slides
q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the

q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the

q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the new currency of the digital world . World Economic Forum 1 Problem: access to big data Data accessed & analyzed Data in organizations

543 views • 30 slides
Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

426 views • 26 slides
Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What

Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What

Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What data do I use most? Session Target: After todays session, you will be able to: 1. Insert/include data opportunities into PL models. 2.

710 views • 12 slides
Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Why prepare the data? Data Preparation Discretization Data cleaning (Data pre-processing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare Data?

426 views • 17 slides
Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer

Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer

Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer Course overview Collect and analyze IoT data Gather data API Endpoints Data Streams Visualize data Combine datasets Detect patterns ML Model

756 views • 27 slides
Modeling Data the different views on Data Mining Views on Data Mining Fitting the data

Modeling Data the different views on Data Mining Views on Data Mining Fitting the data

Modeling Data the different views on Data Mining Views on Data Mining Fitting the data Density Estimation Learning being able to perform a task more accurately than before Prediction use the data to predict future data

517 views • 29 slides
Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction

Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction

Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction Operations on data are central to most solutions Think abstractly about data and its management Typically need to Organize data Add data Remove

756 views • 30 slides
Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper

Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper

Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper by Hadley Wickham, PhD Formalize the way we describe the shape of data Gives us a goal when formatting our data Standard way to organize data

462 views • 27 slides
Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is Big Data Analytics? Data Future Data Past Data Present 1 Big Data Trends Data Driven Organization Other Executives 5% 8% Operations Data

620 views • 22 slides
Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may

Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may

Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may not always come in 1 huge le 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have new data for each

555 views • 31 slides
Data Collection and HIVe Current Data Collection For those collecting data, you are use to

Data Collection and HIVe Current Data Collection For those collecting data, you are use to

Data Collection and HIVe Current Data Collection For those collecting data, you are use to the Excel data collection sheets All Ryan White Funded entities will be required to report client level data Non-Medical Case Management

266 views • 13 slides

More recommend