CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 - PowerPoint PPT Presentation

Dec 01, 2022 •644 likes •822 views

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 ADMINISTRIVIA - Assignment 1: Due Oct 1 - Sign up for Project meetings - Group updates MapReduce GFS BigTable BORG: WORKLOAD Long-running services (should never go down) Batch

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018
ADMINISTRIVIA - Assignment 1: Due Oct 1 - Sign up for Project meetings - Group updates
MapReduce GFS BigTable
BORG: WORKLOAD Long-running services (should “never” go down) Batch jobs: few seconds to a few days
BORG CONCEPTS Users submit jobs Each job is one or more tasks All tasks that run the same program (binary) Each job runs in one Borg cell
JOB DESCRIPTION
JOB PROPERTIES Name Constraints Properties - Resource requirements - No slots! - Static Binaries
JOB LIFECYLE
QUOTAS, PRIORITIES, BNS Priority High priority can preempt lower priority Quotas Used for admission control Infinite quota at priority zero Service Discovery using BNS
ARCHITECTURE
MASTER, Borglet BorgMaster Single Leader, five-ways replicated Paxos group – using Chubby locks Borglet Daemon on each machine Borgmaster pulls updates from Borglets Health checks used to detect failures
SCHEDULER - Feasibility checking pass, Scoring pass - Task cache (static binaries) - Scalability - Split master into multiple processes - Use replicas for communication - Randomize machines used for scoring …
UTILIZATION: CELL COMPACTION
REQUEST SIZE: NO SWEET SPOT
RECLAMATION
LESSONS, DISCUSSION - Jobs are restrictive, Allocs are useful - IP address per container - Kernel of distributed operating system
QUESTIONS / DISCUSSION ?

Recommend

BSD Data Systems Report to the School Board, April 2019 Data-Driven ESSA Best Practices Data

BSD Data Systems Report to the School Board, April 2019 Data-Driven ESSA Best Practices Data Systems Coordinator Data Systems Coordinator Role Data that is accurate, relevant, actionable, accessible. District systems and central office

512 views • 17 slides

Today Data Acquisition Systems Many embedded systems measure quantities from Data

Today Data Acquisition Systems Many embedded systems measure quantities from Data acquisition the environment and turn them into bits Digital filters and signal processing These are data acquisition systems (DAS) Filter

522 views • 9 slides

DLS VERIFICATION TESTING FIELD TESTS TO CHECK DLS OPERATIONS DATA LOGGER SYSTEMS DATA LOGGER

DLS VERIFICATION TESTING FIELD TESTS TO CHECK DLS OPERATIONS DATA LOGGER SYSTEMS DATA LOGGER SYSTEMS Data LOGGER SYSTEMS (DLS) ARE ELECTRONIC SYSTEMS USED TO: 1. MONITOR MATERIAL APPLICATION CONDITIONS i.e. ambient temperatures 2. PROVIDE REAL

492 views • 23 slides

Big Data, Big Problem Data-intensive systems are highly complex Manipulate large-scale

Towards Big Data Provenance Daniel Deutch Blavatnik School of Computer Science, Raymond and Beverly Sackler Faculty of Exact Sciences Tel Aviv University cartoon by T. Gregorius Big Data, Big Problem Data-intensive systems are highly

358 views • 6 slides

Data Centers & Co-designed Distributed Systems A Data Center Inside a Data Center Data

Data Centers & Co-designed Distributed Systems A Data Center Inside a Data Center Data center 10k - 100k servers: 250k 10M cores 1-100PB of DRAM 100PB - 10EB storage 1- 10 Pbps bandwidth (>> Internet) 10-100MW power - 1-2% of

911 views • 54 slides

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu http://manos.athanassoulis.net/classes/CS591 Today big data I want you to speak up! [and you can always interrupt me] data-driven world data systems which are the

860 views • 54 slides

geotools: Exporting cartography data from Stata to GIS systems Sergiy Radyakin

Introduction Earlier products Data sources Data transformations Data export Output Notes 2018 Canadian Stata Conference Morris J. Wosk Centre for Dialogue, Vancouver, BC geotools: Exporting cartography data from Stata to GIS systems

507 views • 37 slides

File Systems 1. Organizing data into files and directories and supporting primitives to

CSE 421/521 - Operating Systems File Systems Fall 2011 Provides organized and efficient access to data on secondary storage: Lecture - XVIII File Systems 1. Organizing data into files and directories and supporting primitives to

345 views • 6 slides

Big Data Systems Big Data Parallelism Huge data set crawled documents, web request logs,

Big Data Systems Big Data Parallelism Huge data set crawled documents, web request logs, etc. Natural parallelism: can work on different parts of data independently image processing, grep, indexing, many more Challenges

708 views • 55 slides

ZODIAC DATA SYSTEMS ZODIAC AIRCRAFT SYSTEMS ITU Mission (from Y. Henri presentation - 18/09/14)

ZODIAC DATA SYSTEMS ZODIAC AIRCRAFT SYSTEMS ITU Mission (from Y. Henri presentation - 18/09/14) ZODIAC AIRCRAFT SYSTEMS This document is the property of ZODIAC DATA SYSTEMS. It cannot be duplicated or distributed without expressed written

744 views • 38 slides

Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data

Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data Centric Systems and Networking Emergence of Big Data Shift of Communication Paradigm From end-to-end to data centric Data as

767 views • 17 slides

Systems for Data Science Marco Serafini COMPSCI 532 Lecture 1 Course Structure

Systems for Data Science Marco Serafini COMPSCI 532 Lecture 1 Course Structure Fundamentals you need to know about systems Caching, Virtual memory, concurrency, etc Review of several Big-data systems Learn how they

428 views • 31 slides

Follow the data! Algorithms and systems for responsible data science Julia Stoyanovich Drexel

Follow the data! Algorithms and systems for responsible data science Julia Stoyanovich Drexel University & Princeton CITP NYC Algorithmic Transparency Law 1/11/2018 Int. No. 1696-A: A Local Law in relation to automated decision systems

812 views • 49 slides

CSE 232A Database System Implementation Arun Kumar Topic 8: Data Systems for ML Workloads

CSE 232A Database System Implementation Arun Kumar Topic 8: Data Systems for ML Workloads Book: Data Management in ML Systems by Morgan & Claypool Publishing 1 Big Data Systems Parallel RDBMSs and Cloud-Native RDBMSs

552 views • 21 slides

File Systems: All data structures are cached for better performance Create a new file

File Systems: Consistency Issues What about Multiple Updates? File systems maintain many data structures Several file system operations update multiple data structures Free list/bit vector Directories Examples: File headers and inode

257 views • 3 slides

How Complex Systems Thinking Can Tame Big Data: The Limits of Data-Centric Inference (and

Research School 2014 Predictive modeling: from data to models 29-31 October, Mtopole, Toulouse How Complex Systems Thinking Can Tame Big Data: The Limits of Data-Centric Inference (and Math Analysis) and Usefulness of Agent-Based

708 views • 51 slides

Lessons and Predictions from 25 Years of Parallel Data Systems Development PARALLEL DATA STORAGE

Lessons and Predictions from 25 Years of Parallel Data Systems Development PARALLEL DATA STORAGE WORKSHOP SC11 BRENT WELCH DIRECTOR, ARCHITECTURE OUTLINE Theme Architecture for robust distributed systems Code structure Ideas

508 views • 37 slides

Software languages for data exchange systems L. Thomas van Binsbergen 1 1 Centrum Wiskunde &

Software languages for data exchange systems L. Thomas van Binsbergen 1 1 Centrum Wiskunde & Informatica l.t.van.binsbergen@cwi.nl April 2020 Section 1 Norm-aware, distributed software systems Regulated data exchange : Data exchange systems

725 views • 51 slides

Elasecutor: Elastic Executor Scheduling in Data Analytics Systems Libin Liu , Hong Xu City

Elasecutor: Elastic Executor Scheduling in Data Analytics Systems Libin Liu , Hong Xu City University of Hong Kong ACM Symposium on Cloud Computing 2018 1 Data Analytics Systems Various workloads running in data analytics systems

844 views • 69 slides

Data Systems that are Easy to Design, Tune and Use Stratos Idreos applications api/sql

Data Systems that are Easy to Design, Tune and Use Stratos Idreos applications api/sql algorithms/operators cpu data data data memory hierarchy data system kernel Stratos Idreos design space it all starts with how we store data every

1.56k views • 117 slides

Files and File Systems files: persistent, named data objects data consists of a sequence of

File Systems 1 Files and File Systems files: persistent, named data objects data consists of a sequence of numbered bytes file may change size over time file has associated meta-data examples: owner, access controls, file type,

459 views • 32 slides

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 ADMINISTRIVIA - Assignment 1 -

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 ADMINISTRIVIA - Assignment 1 - Projects - Piazza MOTIVATION Storing large amounts of semi-structured data - Traditionally done using database systems Varied processing needs - low

388 views • 20 slides

File Systems: Consistency Issues 1 File Systems: Consistency Issues File systems maintain many

File Systems: Consistency Issues 1 File Systems: Consistency Issues File systems maintain many data structures Free list/bit vector Directories File headers and inode structures Data blocks All data structures are cached for better

357 views • 14 slides

on Astrophysical Data Processing Heterogeneous Many-Core Systems Theodore Kisner, LBNL

on Astrophysical Data Processing Heterogeneous Many-Core Systems Theodore Kisner, LBNL Thursday, December 16, 2010 Astrophysical Data Processing Data Acquisition Data Manipulation / Calculations Data Indexing / Storage Data Selection /

953 views • 11 slides