management cdlm
play

Management (CDLM) for Petascale Projects Arun Jagatheesan - PowerPoint PPT Presentation

Jan 27, 2023 •407 likes •603 views

Collaborative Data Life-cycle Management (CDLM) for Petascale Projects Arun Jagatheesan iRODS.org, DICE, SDSC/UCSD Agenda Introductions LSST as use case CDLM Attributes of CDLM History behind the story MDAS (Massive Data

  1. Collaborative Data Life-cycle Management (CDLM) for Petascale Projects Arun Jagatheesan iRODS.org, DICE, SDSC/UCSD

  2. Agenda • Introductions • LSST as use case • CDLM • Attributes of CDLM

  3. History behind the story • MDAS (Massive Data Analysis System) • Support data-intensive applications that manipulate very large data sets by building upon object-relational database technology and archival storage technology • 1995 by DARPA • SDSC SRB (Storage Resource Broker) • iRODS • Flexible license for our community • Flexible rules for users • Flexible data management

  4. My role in iRODS Community • Large-scale usage and adoption of iRODS • Research and Analysis of large-scale use-cases • Design requirements for large-scale users • Consult on iRODS-based storage infrastructure • Community Growth • Tutorials, dissemination • iROD-Chat (2006), SRB-Chat (2003) • Academic and Industrial users

  5. Large Scale Synoptic Survey • Survey entire sky every 3 nights • Dark Energy, Dark Matter, Near Earth Asteroids, and more • World’s largest digital camera (3 billion pixels) • Images 3000 times wider than Hubble • Data from Chile to US and rest of the world • 15 TB/night, over hundred(s) petabytes • www.youtube.com/watch?v=LtMJ_WwvBb8

  6. Data Products • Releases • Cataloged database • Provenance Info QuickTime™ and a • Metadata TIFF (Uncompressed) decompresso are needed to see this picture. • Processed Data Sets • Raw Images

  7. LSST Data Infrastructure Layout QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompr are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture.

  8. LSST Data Train QuickTime™ and a and iRODS TIFF (Uncompressed) decompressor are needed to see this picture. /file1..10.fits /file1..10.fits QuickTime™ and a /nobel.event TIFF (Uncompressed) decompressor are needed to see this picture. /file1..10.fits /catalog1.db /catalog1.db UK or IN2P3 /file1..10.fits /file1..10.fits /catalog1.db /catalog1.db

  9. LSST CDLM Problem Statement • LSST data-lifecycle management infrastructure for: • Performance oriented data storage sub-systems • Capacity oriented data storage sub-systems • Data (usage oriented) distribution networks • [Provenance and archive storage systems] • Confluence of three major storage dimensions • HPC data processing (pipelines to produce our data) • Datacenter sharing (data centers that host our data) • Data delivery and distribution (usage of our data)

  10. CDLM • Collaborative Data Lifecycle Management • Multiplexing of a single data life-cycle amongst more than one autonomous partner • Attributes of data-lifecycle is shared • Varying levels of autonomy and inter- dependence

  11. Multiplexing a Data Life-cycle • Data Creation (Raw data) • Data Processing (Derived data) • Data Analysis (Data warehouse, ..) • Data Namespace • Data Dissemination • Data Provenance • Data Archival

  12. Levels of Collaboration • Collaboration on Data Life-cycle not necessarily mean collaboration of businesses • Some types of CDLM • Symbiotic - All partner businesses benefit from CDLM • Neutral - No effect on businesses due to CDLM • Competitive - partners of CDLM are actually competitors of the resulting business process (forced to have a common platform to compete) • Hybrid - Multiple or transient partner relationships

  13. Autonomy & Inter-dependence at right levels for CDLM to work

  14. LSST Data Layout QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompr are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture.

  15. ALMA data flow QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) d are needed to see this QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) de are needed to see this

  16. LSST SC-2008 Prototype QuickTime™ and a TIFF (Uncompressed) d are needed to see this QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompre are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decom are needed to see this pictu QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

  17. CDLM Infrastructure Design • Requirements, Expectations and Performance Management • Minimize dependencies (without affecting cost) • Reduce individual autonomy into hierarchical groups (that can remain autonomous) • Hierarchical rules and community rules

  18. iRODS enabling CDLM • Global Namespace • Resource allocation and service levels as policies/rules • Hierarchical rules and access controls • Highly Flexible System

  19. Similar projects? Let’s talk • The power of the community • Not necessarily “large” scale • Symbiotic • arun@diceresearch.org

Recommend

Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval

Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval

Why ranked retrieval? Term frequency tf-idf weighting The vector space model Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval CdLM in Computer Science University of Rome Tor Vergata Derived

769 views • 51 slides
MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of

MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of

MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of the limits of an organization's resources, such as its labor force, manufacturing and office space, technology and equipment, raw materials, and

617 views • 26 slides
Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a

Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a

Moreno Baricevic Stefano Cozzini CNR-IOM DEMOCRITOS Trieste, ITALY Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a pool of users and a pool of resources, then what? some software that controls

538 views • 22 slides
Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op.

Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op.

Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op. Risk Management Board of Enterprise Risk Senior Management Committees Directors Management Credit Risk Cross-Border Exposure Country Rating

453 views • 6 slides
Configuration management Configuration management Configuration management Configuration

Configuration management Configuration management Configuration management Configuration

Configuration management Configuration management Configuration management Configuration management Configuration management Field of management that focuses on establishing and maintaining consistency of a systems attributes

647 views • 6 slides
Medical Organization Needs SOA Patients Service Oriented Architecture Finance Doctors Data

Medical Organization Needs SOA Patients Service Oriented Architecture Finance Doctors Data

Management of Patient Records Management of Medical Records Management of Services Price management Relations with Insurance Company Financial Management and Accounting Medical Supplies Management Auxiliary

952 views • 9 slides
What is Risk Management? What is Risk Management? A (great) management tool Focus

What is Risk Management? What is Risk Management? A (great) management tool Focus

Proudly Presents How To Establish Functional Risk Management in Your Public Entity R. J oy J ackson, City of London Tina Gardiner, Region of York RIMS, Ottawa, 2011 What is Risk Management? What is Risk Management? A (great)

463 views • 24 slides
Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main

Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main

Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main reason to manage your own home is cost savings. This may be a valid enough reason to undertake self management. However, there are some items to

393 views • 17 slides
Change Management Change Management Change Management Appreciate the balance within Change

Change Management Change Management Change Management Appreciate the balance within Change

Slide 2 Purpose of this Session Objectives Understanding of Change Management principles within RBS Understand why CM is important / necessary Change Management Change Management Change Management Appreciate the balance within

250 views • 4 slides
Financial Management Review of Financial Management Financial Management System Internal

Financial Management Review of Financial Management Financial Management System Internal

Financial Management Review of Financial Management Financial Management System Internal Controls Accuracy of Report Information Program Income Salaries And Wages OMB Circular A-133 Single Audits Financial Management

461 views • 12 slides
Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value

Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value

Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value Management Team State Value Management Engineer Jessica Kuse, PE, CPM Value Management Value Management Value Management Value Management Program

922 views • 44 slides
IT Management, Simplified Real-&me IT management solu&ons for the

IT Management, Simplified Real-&me IT management solu&ons for the

IT Management, Simplified Real-&me IT management solu&ons for the new speed of business Product Overview Who we are A Real-Time IT Management Company Enterprise IT management software

370 views • 19 slides
MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management

MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management

MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management Studies Indian Institute of Technology Madras Chennai 600 036 E-mail: mtm@iitm.ac.in WHAT IS MANAGEMENT ? 1.Field of Study - Management principles,

286 views • 16 slides
Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the

Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the

Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the entire lifecycle of cryptographic keys and other keying material. Basic key management guidance is provided in [SP800-21]. A single item of keying

522 views • 18 slides
Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum

Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum

Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum Chicago, IL Jeff Gerson Al Derosa, CFA Business Practice Mgmt Fixed Income Kurt Meyer Paul Davis Equities Marketing Running a Successful PM

772 views • 65 slides
Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate

Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate

Bu ff er Management Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate students who faced challenges with setting up the virtual machine and / or getting familiar with C ++ , we are bumping up the number of

973 views • 46 slides
Management & Leadership Professor Kerry Goulston Management & Leadership MANAGEMENT

Management & Leadership Professor Kerry Goulston Management & Leadership MANAGEMENT

2/28/2014 Management & Leadership Professor Kerry Goulston Management & Leadership MANAGEMENT - Make sure the organisation runs well - Steady continuous improved performance as a whole LEADERSHIP - Involves vision and

148 views • 4 slides
ASSET MANAGEMENT December 7, 2018 WHAT IS ASSET MANAGEMENT? ASSET MANAGEMENT IS A PROCESS TO

ASSET MANAGEMENT December 7, 2018 WHAT IS ASSET MANAGEMENT? ASSET MANAGEMENT IS A PROCESS TO

BREAKING THROUGH: OVERCOMING THE WATER FUNDING CHALLENGE ASSET MANAGEMENT December 7, 2018 WHAT IS ASSET MANAGEMENT? ASSET MANAGEMENT IS A PROCESS TO ENSURE THAT THERE IS SUFFICIENT INVESTMENT IN AND PLANNED MAINTENANCE, REPAIR AND

627 views • 7 slides
Overview of this weeks material 2 Procurement management is one of the nine major project

Overview of this weeks material 2 Procurement management is one of the nine major project

Overview of this weeks material 2 Procurement management is one of the nine major project management knowledge areas according to PMI Project Integration management, scope management, time management, cost management, quality

375 views • 6 slides
Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on

Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on

Memory Management Lecture 5: Memory Management 1 / 54 Memory Management Administrivia Assignment 1 is due on September 7th @ 11:59pm 2 / 54 Memory Management Poll https: // www.strawpoll.me / 20863490 3 / 54 Memory Management Recap

952 views • 54 slides
Portfolio management 1 Portfolio management and investment banking Asset management and

Portfolio management 1 Portfolio management and investment banking Asset management and

Portfolio management 1 Portfolio management and investment banking Asset management and trading constitute the other major activity of IB, alongside investment banking (i.e. corporate finance) For large banks, asset management and

493 views • 45 slides
Lecture Notes: Message Management 1 Slide 1: Message Management Message Management A critical

Lecture Notes: Message Management 1 Slide 1: Message Management Message Management A critical

Lecture Notes: Message Management 1 Slide 1: Message Management Message Management A critical aspect of successful advocacy is the ability to create and deliver a message. While your passion matters greatly, your coalitions make the work

487 views • 15 slides
Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

4/12/2017 Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign physical memory to processes 5B. Simple Memory Allocation explicit requests: malloc (sbrk) implicit: program loading, stack

205 views • 8 slides
Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign

4/14/2018 Memory Management Memory Management 5A. Memory Management and Address Spaces 1. allocate/assign physical memory to processes 5B. Memory Allocation explicit requests: malloc (sbrk) implicit: program loading, stack extension

555 views • 9 slides

More recommend