Data At Rest Data In Motion! A Lambda Architecture Overview When - PowerPoint PPT Presentation

Dec 03, 2023 •181 likes •377 views

Data At Rest Data In Motion! A Lambda Architecture Overview When Things Go Wrong http://xkcd.com/327/ Fault T olerance !!!! Fault T olerance Developer Software Hardware Data Collection Three T ypes Of Data Streams Structured

Data At Rest … Data In Motion! A Lambda Architecture Overview
When Things Go Wrong http://xkcd.com/327/ Fault T olerance !!!!
Fault T olerance Developer Software Hardware
Data Collection Three T ypes Of Data Streams Structured (Databases ...) Semi Structured (JSON, XML, XAML ...) UnStructured (Blogs, E-Mails, Log Files ...)
Lambda Architecture T o The Rescue !!
Lambda Architecture — Requirements Fault-tolerant against both hardware failures and human errors Support variety of use cases that include low latency querying as well as updates Linear scale-out capabilities Extensible, so that the system is manageable and can accommodate newer features easily
Lambda Architecture IMMUTABLE PRECOMPUTE MASTER DATA VIEWS BATCH RECOMPUTE View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT STREAM VIEWS REAL-TIME INCREMENT
Lambda Architecture IMMUTABLE PRECOMPUTE BATCH LAYER MASTER DATA VIEWS BATCH RECOMPUTE SERVING LAYER View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT SPEED LAYER STREAM VIEWS REAL-TIME INCREMENT
Lambda Architecture - Layers Batch Layer Managing the master data set, an immutable, append only set of raw data. Pre computing arbitrary query functions, called batch views.
Lambda Architecture - Layers Serving Layer Indexes batch views so that they can be queried in ad hoc with low latency. Merges and reconciles batch and real time views.
Lambda Architecture - Layers Speed Layer Accommodates all requests that are subject to low latency requirements. Using fast and incremental algorithms, deals with recent data only.
Lambda Architecture - Reconciliation Data absorbed into Batch Views Not yet absorbed Time
Lambda Architecture - Reconciliation Now Just a few hours of data Data absorbed into Batch Views Not yet absorbed Time
Lambda Architecture - Immutable Data + Views Times Airpor Flight Action tamp t 2015- DUB EL123 take- 01- off 01T10: 00:0 2015- HEL SA45 take- 01- off 01T10: 05:0 2015- AMS BA99 take- 01- off 01T10: Immutable Master Dataset 07:0 2015- LHR LH17 landin 01- g
Lambda Architecture - Immutable Data + Views Timestamp Airport Flight Action air borne: 2307 Map Reduce 2015-01- DUB EL123 take-off 01T10:00:0 air borne per airline: Map Reduce 2015-01- HEL SA45 take-off 01T10:05:0 Airline 2015-01- AMS BA99 take-off Map Reduce airport load: 01T10:07:0 Airport Planes 2015-01- LHR LH17 landing 01T10:09:0 2015-01- CDG AF03 landing SAS 01T10:10:0 Immutable Master Dataset 2015-01- FCO AZ501 take-off AMS 44 01T10:11:0 BA LHR 69
Lambda Architecture - Implementation IMMUTABLE PRECOMPUTE MASTER DATA VIEWS BATCH RECOMPUTE View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT STREAM VIEWS REAL-TIME INCREMENT
Lambda Architecture - Implementation Hadoop HDFS Apache Hive BATCH RECOMPUTE HBase HBase HBase NEW DATA ● Spark SQL ● R MERGE STREAM ● Presto Storm Storm Storm Bolt Bolt Bolt Apache Kafka Apache Spark REAL-TIME INCREMENT

Recommend

FROM 00s TO 20s: FROM REST ful to gRPC Gianfranco Reppucci @giefferre Data Engineer WHAT

FROM 00s TO 20s: FROM REST ful to gRPC Gianfranco Reppucci @giefferre Data Engineer WHAT THIS TALK IS ABOUT BEFORE REST S O A P Common Object Request Simple Object Broker Architecture Access Protocol REST: REpresentational State

792 views • 44 slides

Motion Cues for Illustration of Skeletal Motion Capture Data Simon Bouvier-Zappa Victor

Motivation Motion Capture Hierarchy Motion Cues Results Video Conclusion Motion Cues for Illustration of Skeletal Motion Capture Data Simon Bouvier-Zappa Victor Ostromoukhov Pierre Poulin LIGUM, Dept. I.R.O Universit de Montral

961 views • 75 slides

Outline What is ReST ? Constraints in ReST REST Architecture Components Features of

Outline What is ReST ? Constraints in ReST REST Architecture Components Features of ReST applications Example of requests in REST & SOAP Complex REST request REST Server response Real REST examples REST Design

603 views • 34 slides

DDD & REST Domain-Driven APIs for the web Oliver Gierke / olivergierke 2 Background

DDD & REST Domain-Driven APIs for the web Oliver Gierke / olivergierke 2 Background 3 Spring Data REST Spring Data Spring HATEOAS Repositories & Hypermedia Aggregates for Spring MVC 4 REST CRUD via HTTP 5

1.17k views • 55 slides

Topic 6 Generic Data Structures "Get your data structures correct first, and the rest of

Topic 6 Generic Data Structures "Get your data structures correct first, and the rest of the program will write itself." - David Jones Back to our Array Based List Started with a list of ints Don't want to have to write a new

202 views • 18 slides

Linked Open Data data.slub-dresden.de Linked Open Usable Data data.slub-dresden.de schema.org

Linked Open Data data.slub-dresden.de Linked Open Usable Data data.slub-dresden.de schema.org schema:CreativeWork schema:Person schema:Organization schema:Event schema:Place schema.org WIP ;) REST API with JSON-LD REST API with

531 views • 17 slides

How to get Twitter Data from the Twitter REST and Streaming API David M. Beskow Carnegie Mellon

How to get Twitter Data from the Twitter REST and Streaming API David M. Beskow Carnegie Mellon University 29 January 2018 Collecting Data on the Web in General 1. What platform should I use? 2. Should I collect everything? 3. How much

397 views • 20 slides

REST Web-based APIs REST Representational State Transfer Style of web software

REST Web-based APIs REST Representational State Transfer Style of web software architecture that simplifies application Not a standard, but a design pattern REST Take all resources for web application (data, files, functions)

722 views • 22 slides

Encrypted File Systems Don Porter CSE 506 Goals Protect confidentiality of data at rest

Encrypted File Systems Don Porter CSE 506 Goals Protect confidentiality of data at rest (i.e., on disk) Even if the media is lost or stolen Protecting confidentiality of in-memory data much harder Continue using file system features

227 views • 19 slides

Algorithms and Data Structures for Embedded Network Data Minkyoung Cho, David Mount, and Eunhui

Introduction LSE Cost computation Nets and net trees Incremental motion Maintaining nets under motion End Matter Algorithms and Data Structures for Embedded Network Data Minkyoung Cho, David Mount, and Eunhui Park Department of Computer

630 views • 39 slides

Greg Neiheisel CTO Astronomer Data Engineering Platform Streaming data Data pipelines Code

Greg Neiheisel CTO Astronomer Data Engineering Platform Streaming data Data pipelines Code first ETL Early Priorities Quick prototyping Get data in motion Ease of scale Astronomer V1 Lambda + API Gateway Cloudwatch for Monitoring

655 views • 26 slides

Artificial motion data for navigation sensors of a submarine running in periscope depth below wave

Artificial motion data for navigation sensors of a submarine running in periscope depth below wave surface 2018.06.28. Hyeon Kyu Yoon, Young-Ho Park, Juwon Seo (CWNU) Youngbum Park, Chanju Park (ADD) Artificial motion data of submarine in wave

863 views • 21 slides

AND NEWTONS LAWS Law 1 Every body perseveres in its state of rest, or of uniform

AND NEWTONS LAWS Law 1 Every body perseveres in its state of rest, or of uniform motion in a right line, unless it is compelled to change that state by forces impress'd thereon Law II The alteration of motion is

263 views • 25 slides

Order of Presentations Team Topic 205 IoT Data Flows - Save the ISS! 201 Risks & REST

Order of Presentations Team Topic 205 IoT Data Flows - Save the ISS! 201 Risks & REST with CoAP 202 LoRaWAN - The backbone of LoRa Networks 208 IoT orchestration with RabbitMQ 203 Sending a picture from a Raspberry Pi/Arduino via

1.28k views • 123 slides

Understanding Data Motion in the Modern HPC Data Center Glenn K. Lockwood Shane Snyder Suren

Understanding Data Motion in the Modern HPC Data Center Glenn K. Lockwood Shane Snyder Suren Byna Philip Carns Nicholas J. Wright - 1 - Scientific computing is more than compute! Tape Tape Tape Tape Tape Tape Tape Tape GW Tape

360 views • 23 slides

Training of Deep Bidirectional RNNs for Hand Motion Filtering via Multimodal Data Fusion Soroosh

Introduction HMFP-DBRNN Data Fusion Conclusion Training of Deep Bidirectional RNNs for Hand Motion Filtering via Multimodal Data Fusion Soroosh Shahtalebi , S. Farokh Atashzar , Rajni V. Patel , and Arash Mohammadi

201 views • 18 slides

Physics, Big Data Analysis and Philosophy (and all the rest) Wolfgang Rhode ... eritis sicut

Physics, Big Data Analysis and Philosophy (and all the rest) Wolfgang Rhode ... eritis sicut Deus ... Platos Cave Analogy Are true conclusions possible? Within the world of logic ? Yes Within the world of mathematics ? Yes Within

800 views • 60 slides

Computer Vision by Learning: Motion in Action Jan van Gemert, UvA 2 Motion and perceptual

Computer Vision by Learning: Motion in Action Jan van Gemert, UvA 2 Motion and perceptual organization Even impoverished motion data can evoke a strong percept 3 Motion and perceptual organization Even impoverished motion

594 views • 44 slides

Motion capture Captures the subtlety of human motion Keyframing difficult and not as

Motion capture Captures the subtlety of human motion Keyframing difficult and not as successful Cons: We can use only prerecorded material (with certain adaptations)-limiting Data valid only for same scale avatar-need for

772 views • 56 slides

Programming With Data One-Slide Summary A list is a data structure , a way of storing and

Programming With Data One-Slide Summary A list is a data structure , a way of storing and organizing data. cons creates a pair of two values. car and cdr (or first and rest ) extract the first and second elements of a cons pair. A

205 views • 8 slides

Data-Driven Animation Full-body animation Skin deformation Facial animation Motion

Data-Driven Animation Full-body animation Skin deformation Facial animation Motion Representation A pose is represented by root translation, root orientation, and joint angles q = ( q 1 , , q n ) Motion consists of a

368 views • 24 slides

Linked List Implementation Data-structure-palooza Checkout LinkedLists project from SVN

Linked List Implementation Data-structure-palooza Checkout LinkedLists project from SVN Understanding the engineering trade-offs when storing data Efficient ways to store data based on how well use it The main theme for the rest of

696 views • 24 slides

REST API What it means to be RESTful What is REST REST stands for REpresentational State

REST API What it means to be RESTful What is REST REST stands for REpresentational State Transfer It is neither a standard nor a protocol REST is not HTTP , although because of HTTPs prevalence and popularity the two tend to go

708 views • 11 slides

...or is it? Physics Ilenia Salvadori Data In Motion Consulting GmbH 1 About us: Founded

STORY OF A JOURNEY: OSGi is no Astroparticle ...or is it? Physics Ilenia Salvadori Data In Motion Consulting GmbH 1 About us: Founded in 2010; Located in Jena/Thuringia Germany; Consulting, Independent RnD, Development,

280 views • 27 slides