introduction to introduction to datawarehousing
play

Introduction to Introduction to Datawarehousing Umberto Nanni - PowerPoint PPT Presentation

Oct 12, 2023 •455 likes •699 views

D IPARTIMENTO DI I NGEGNERIA INFORMATICA AUTOMATICA E GESTIONALE A NTONIO R UBERTI Master of Science in Engineering in Computer Science (MSE-CS) (MSE-CS) Seminars in Software and Services for the Information Society Umberto Nanni Introduction

  1. D IPARTIMENTO DI I NGEGNERIA INFORMATICA AUTOMATICA E GESTIONALE A NTONIO R UBERTI Master of Science in Engineering in Computer Science (MSE-CS) (MSE-CS) Seminars in Software and Services for the Information Society Umberto Nanni Introduction to Introduction to Datawarehousing Umberto Nanni Seminars of Software and Services for the Information Society 1

  2. Business Intelligence Architecture management system system management management KPI DSS MKT CRM HR system … data mining reporting OLAP goals results Datamart-1 Datamart-2 Datamart-3 Datawarehouse ETL systems operational system external service ERP internet / data systems systems extranet sources operational system Umberto Nanni Seminars of Software and Services for the Information Society 2

  3. What is Data Warehousing Collection of methods, technologies and tools to assist the “knowledge worker” (manager, analyst) to conduct data analysis aimed at supporting decision-making and/or improving the management of information assets Umberto Nanni Seminars of Software and Services for the Information Society 3

  4. What is a Data Warehouse A data warehouse is a collection of data • integrated (far beyond the organization) • consistent (despite the heterogeneous origin) • focused (an interest area is defined) • historical (over a consistent timeframe) • permanent (never delete your data!) Umberto Nanni Seminars of Software and Services for the Information Society 4

  5. Purpose of a Data Warehouse A Data Warehouse helps (allows) you: to take decisions • to identify and interpret phenomena • to make predictions about the future to make predictions about the future • • to control a complex system • Umberto Nanni Seminars of Software and Services for the Information Society 5

  6. Value and quantity of information logistics sales marketing BD prices prices competitors competitors value value strategic information reports $$$ $$$ $ selected selected information primary information sources quantity Umberto Nanni Seminars of Software and Services for the Information Society 6

  7. OLTP & OLAP OLTP - On-Line Transaction Processing – realm of (write and / or read) transactions, recovery, consistency consistency – many, fast and frequent operations – high level of concurrency – access to a small amount of data – on-the-fly data update OLAP - On-Line Analytical Processing – read only read only – few operations – low level of concurrency – access to huge amounts of data – historical but essentially static data Umberto Nanni Seminars of Software and Services for the Information Society 7

  8. Separation between: Operational Database & Data Warehouse • different computational load • different needs: • different needs: – DB: dynamic data, asynchronous updates – DW: static data, periodic updates • integration with business activity: – DB: supporting operations (focused, timely) – DW: supporting decisions (descriptive, historical) – DW: supporting decisions (descriptive, historical) • data collection: – DB: minimal – DW: maximal Umberto Nanni Seminars of Software and Services for the Information Society 8

  9. Two issues with different perspectives • Data redundancy – OLTP (DB): to avoid, bringing to inconsistency – OLTP (DB): to avoid, bringing to inconsistency and/or inefficiency on updates – OLAP (DW): redundancy avoids recomputation and shorten response time • Indexing – OLTP (DB): good when you search – bad when you – OLTP (DB): good when you search – bad when you update... you need some trade-off – OLAP (DW): the more, the best Umberto Nanni Seminars of Software and Services for the Information Society 9

  10. Some Data Warehouse Systems • Oracle 12 • IBM InfoSphere Warehouse • IBM InfoSphere Warehouse • Microsoft SQL-Server 2012 – Analysis Services • Sybase IQ • Hyperion (bought by Oracle) • Teradata (division of NCR) • Teradata (division of NCR) • Netezza – Cognos (bought by IBM) • Business Objects (bought by SAP) Umberto Nanni Seminars of Software and Services for the Information Society 10

  11. A comparison by Gartner Donald Feinberg, Mark A. Beyer Magic Quadrant for Data Warehouse Database Management Systems Gartner RAS Core Research Note G00173535, 28 January 2010 Umberto Nanni Seminars of Software and Services for the Information Society 11

  12. Architectures for Datawarehousing: issues • separating OLTP & OLAP • scalability • extensibility • security • security • administrability Umberto Nanni Seminars of Software and Services for the Information Society 12

  13. Architecture for Datawarehousing • determined by design choices • determined by / determines the choice of a • determined by / determines the choice of a software system • determines the cost and makes possible future integration (quantitative and / or qualitative) qualitative) • affects the cost of data processing Umberto Nanni Seminars of Software and Services for the Information Society 13

  14. Data Mart Collection of data focused on particular user profile or on particular target analysis on particular target analysis Alternatives: 1. dependent Data Mart: it is a subset and/or an aggregation of data in the primary DW → DM extracted from a DW 2. independent Data Mart: it is a subset and/or an aggregation 2. independent Data Mart: it is a subset and/or an aggregation of data in the operational DB → DW= U i (DM i ), that is, DW is a set of DM 3. hybrid solution, combining 1, 2 Umberto Nanni Seminars of Software and Services for the Information Society 14

  15. DW architecture: 1 Level • there is only an operational DW • virtual DB (no OLTP-OLAP separation) • data coincident with DB operational • difficult integration with other sources data - level 1 middleware (copy of) operational operational DB external sources sources warehouse analysis Umberto Nanni Seminars of Software and Services for the Information Society 15

  16. DW architecture: 2 Levels – dependent DMs • data sources complemented with external sources • running on dedicated software platform • ETL: Extraction, Transformation, Loading • ETL: Extraction, Transformation, Loading • materialization of the DW • materialization of Data Marts data - level 1 data - level 2 ETL oper Data Mart Mart BD BD DW ext Data BD Mart sources feeding warehouse analysis Umberto Nanni Seminars of Software and Services for the Information Society 16

  17. DW architecture: 2 Levels – independent DMs • Data Mart are materialized by feeding • DW = union of DMs • DW = union of DMs data - level 1 data - level 2 ETL oper Data BD BD Mart Mart ext Data BD Mart sources feeding warehouse analysis Umberto Nanni Seminars of Software and Services for the Information Society 17

  18. DW architecture: 3 Levels a level of "reconciled" data (operational data store) is • introduced separation into two phases of ETL activities: separation into two phases of ETL activities: • • 1. extraction / transformation 2. loading data - level 1 data - level 2 data - level 3 ET(L) oper Data BD BD Mart Mart reconcilied DW data ext Data BD loading Mart sources feeding warehouse analysis Umberto Nanni Seminars of Software and Services for the Information Society 18

  19. ETL: Extraction, Transformation, Loading Operational Data, External Data extraction extraction • cleaning - validation - filtering • transformation • Reconciled Data loading • Data Warehouse Umberto Nanni Seminars of Software and Services for the Information Society 19

  20. Extraction • initial extraction: – targeted at the creation of the DW – targeted at the creation of the DW • furter extractions: – static (replaces the whole DW) – incremental – incremental • log • timestamp Umberto Nanni Seminars of Software and Services for the Information Society 20

  21. Cleaning • changing VALUES • duplicates • duplicates • inconsistencies – domain violation – functional dependency violation • null values • misuse of fields • misuse of fields • spelling • abbreviations (not homogeneous) Umberto Nanni Seminars of Software and Services for the Information Society 21

  22. Transformation • changing FORMATS: • misalignment of formats • field overloading • unhomogeneous coding • unhomogeneous coding Umberto Nanni Seminars of Software and Services for the Information Society 22

  23. Loading • Refresh: ex-novo load of the whole DW ex-novo load of the whole DW • Update: differential updates Umberto Nanni Seminars of Software and Services for the Information Society 23

  24. Metadata • internal metadata – concerning the administration of the DW (i.e., sources, transformations, schemas, users, etc..) schemas, users, etc..) • external metadata – interesting for users (e.g., measurement units, possible combinations) • STANDARDs • CWM - Common Warehouse Model (OMG), defined by: – UML (Unified Modeling Language) – UML (Unified Modeling Language) – XML (eXtensible Markup Language) – XMI (XML Metadata Interchange) OMG = Object Management Group: CORBA (Common Object Request Broker Architecture), UML (Unified Modeling Language) , MDA (Model-Driven Architecture) Umberto Nanni Seminars of Software and Services for the Information Society 24

Recommend

Datawarehousing para datos genticos, socioeconmicos y fenotpicos, con visualizacin 3D

Datawarehousing para datos genticos, socioeconmicos y fenotpicos, con visualizacin 3D

Datawarehousing para datos genticos, socioeconmicos y fenotpicos, con visualizacin 3D SciPy 2018 Luciano Serruya Aloisi Pablo Toledo Margalef Universidad Nacional de la Patagonia San Juan Bosco August 31, 2018 1 / 16 Roadmap

384 views • 16 slides
INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION & FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
1 Introduction Introduction Communication System Baseband an adjective that describes

1 Introduction Introduction Communication System Baseband an adjective that describes

Introduction Introduction Physical Layer Mobile network Global ISP Provides the means to transmit bits Basic Concepts of from sender to receiver, Data Transmission Defines the mechanical, electrical, and Home network timing

463 views • 7 slides
Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

INFOGR Computer Graphics Jacco Bikker - April-July 2015 - Lecture 1: Introduction Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical Details Assignments Field Study State of

869 views • 62 slides
Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Virtual and Augmented Reality Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction Who am I? What do I do here? 2 About me Short research bio: - 2014 - finished PhD at FMFI UK - 2014 2016 -

1.3k views • 39 slides
PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience for your presentation. It tells them what your presentation is about, why it is important, and finally, what to listen for in your speech. Get the

505 views • 3 slides
How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

Introduction to Beamer Hafida Benhidour How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer Introduction to Beamer Hafida Benhidour Department of computer science King Saud University November

732 views • 49 slides
Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Shenzhen Cuilu jewelry Co., Ltd

147 views • 14 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

558 views • 18 slides

More recommend