between you and your data
play

Between You and your Data Matt Crawford Fermilab Computing - PowerPoint PPT Presentation

Jul 04, 2023 •204 likes •363 views

Between You and your Data Matt Crawford Fermilab Computing Division Data Movement & Storage ... are the networks LHCOPN USLHCNET Regional Networks Internet2 ESnet Site networks Getting your data AMS USLHCNET

  1. Between You and your Data Matt Crawford Fermilab Computing Division Data Movement & Storage

  2. ... are the networks • LHCOPN • USLHCNET • Regional Networks • Internet2 • ESnet • Site networks

  3. Getting your data AMS USLHCNET CERN CHI PAR NY FNAL BNL DC ESnet Internet2

  4. European side GEANT2 NREN NREN NREN NREN... Euro Euro Euro Euro CERN T1 T1 T1 T1... LHCOPN

  5. Networks international international international connections connections connections Loop off e l Backbone t t a d e SDN Core n S a o l g e a v c e i l h C C New York Sunnyvale IP Core Denver Washington Atlanta Metropolitan DC Area Rings international connections LA Albuquerque international San connections Diego 10-50 Gbps circuits IP core hubs Production IP core international SDN hubs Science Data Network core connections Metropolitan Area Networks Primary DOE Labs or backbone loops for Lab access Possible hubs International connections

  6. Exchange Points ... and many, many others ...

  7. Circuits

  8. Exploiting Circuits • Dynamic path-selection services now being deployed • LambdaStation, Terapaths are site interfaces to: • BRUW, DRAGON, OSCARS, UCLP , ... in the WAN. • LambdaStation client will be activated on FNAL T1 dCache Thursday morning.

  9. Site network – CMS T1

  10. S C 4 C S A 0 6 C S A 0 7

  11. Time to Upgrade!

  12. Making it work • The CMS computing model makes no concession to geography – all T2/T3 centers take data from any T1 centers. • Round-trip time is the enemy of throughput. • Fermilab is committed to making the model work. We view giving good service to Euro- T2s as a condition of requiring good service from Euro-T1s. • Excellent service to US sites is a given.

  13. System factors • Linux has problems. • We are researching, testing, and contributing kernel fixes. • Network research feeds into dCache/SRM (and now, gridFTP) development.

  14. Call us • US-CMS center • Tier-1 center managers (CMS) • Storage developers • Network engineers • OS performance

  15. * T0 CERN 128.142.128.0/17 You can use ATLAS T1 BNL dcdoor0{1,2,3,4,5}.usatlas.bnl.gov CMS T1 FNAL {cmssrm,cmslcgce,cmsuaf,cmswn051.fnal.gov * T1 ASCC osgc01.grid.sinica.edu.tw traceroute to CMS T2 UNL {red,thpc-1}.unl.edu CMS T2 WISC {cmsgrid02,cmssrm}.hep.wisc.edu CMS T2 CALTECH {cit-gatekeeper,cithep59}.ultralight.org check your paths CMS T2 UCSD {osg-gw-2,t2data2}.t2.ucsd.edu CMS T2 Florida {ufloridapg,ufgrid01,srm1}.phys.ufl.edu to other sites. CMS T2 MIT {ce01,se01}.cmsaf.mit.edu CMS T2 Purdue {osg,dcache}.rcac.purdue.edu CMS T2 SPRACE {sprace,spgrid,spdc00}.if.usp.br CMS T2 UERJ prod-frontend.hepgrid.uerj.br CMS T2 ÖAW bauch.oeaw.ac.at * T1 IN2P3 ccsrm.in2p3.fr (Please send me * T1 CNAF * T1 GridKa gridka-dcache.fzk.de additions to this table!) ATLAS T1 NIKHEF ATLAS T1 Nordic * T1 PIC * T1 RAL ATLAS T1 TRIUMF CMS T1 Korea ATLAS Japan * China ATLAS AU * T2 SINP lcg06.sinp.msu.ru * T2 ITEP ceitep.itep.ru * T2 JINR lcgce01.jinr.ru * T2 IHEP ce0001.m45.ihep.su * T2 KI gate.grid.kiae.ru * T2 PNPI cluster.pnpi.nw.ru * T2 FIAN ce1.grid.lebedev.ru * T2 INR grce001.inr.troitsk.ru ATLAS T2 BU atlas.bu.edu ATLAS T2 OU tier2-01.ochep.ou.edu ATLAS T2 UTA atlas.dpcc.uta.edu ATLAS T2 UNM ATLAS T2 LU ATLAS T2 CA-E ATLAS T2 CA-W ATLAS T2 Chicago ATLAS T2 IU ATLAS T2 Harvard CMS T3 VU vampire.accre.vanderbilt.edu

Recommend

Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data

DATA MINING LECTURE 2 What is data? The data mining pipeline What is Data Mining? Data mining is the use of efficient techniques for the analysis of very large collections of data and the extraction of useful and possibly unexpected

2.4k views • 94 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 68 / 91 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4 2

579 views • 8 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 28 / 89 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4

455 views • 23 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 51 / 89 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4

344 views • 17 slides
Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval

Inf1-DA 20102011 III: 68 / 91 Part III Unstructured Data Data Retrieval: III.1 Unstructured data and data retrieval Statistical Analysis of Data: III.2 Data scales and summary statistics III.3 Hypothesis testing and correlation III.4 2

729 views • 24 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and repairing account for about 60% of the work of data scientists." Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides
Data stories with The Pudding So what is data storytelling? Data academia & data science

Data stories with The Pudding So what is data storytelling? Data academia & data science

Data stories with The Pudding So what is data storytelling? Data academia & data science dense complex static/non-interactive often not accompanied by an easy-to-follow narrative Data art beautiful abstract

647 views • 5 slides
Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by

Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by

Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by some geophysical device Most surveys are comprised of linear traverses or transects Typically constant data spacing Perpendicular to

432 views • 12 slides
Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges:

Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges:

Data Collection and Aggregation Data Collection and Aggregation 1 Challenges: data Challenges: data Data type: numerical sensor readings. Rich and massive data, spatially distributed and correlated. Data dynamics: data

714 views • 48 slides
Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data

Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data

DATA Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data Obtaining data Further study THE ROLE OF DATA Data refers to observed facts there are 82 persons in this train the weight of

683 views • 22 slides
DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type

DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type

DataCamp Data Types for Data Science DataCamp Data Types for Data Science Data types Data type system sets the stage for the capabilities of the language Understanding data types empowers you as a data scientist DataCamp Data Types for Data

527 views • 23 slides
q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the

q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the

q.Datum Data Exchange The marketplace for big data Data is the new oil of the Internet and the new currency of the digital world . World Economic Forum 1 Problem: access to big data Data accessed & analyzed Data in organizations

543 views • 30 slides
Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

426 views • 26 slides
Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What

Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What

Prioritizing Data and Purpose of Data Points What data do I have? What data do I trust? What data do I use most? Session Target: After todays session, you will be able to: 1. Insert/include data opportunities into PL models. 2.

710 views • 12 slides
Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Why prepare the data? Data Preparation Discretization Data cleaning (Data pre-processing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare Data?

426 views • 17 slides
Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer

Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer

Introduction to IoT data AN ALYZ IN G IOT DATA IN P YTH ON Matthias Voppichler IT Developer Course overview Collect and analyze IoT data Gather data API Endpoints Data Streams Visualize data Combine datasets Detect patterns ML Model

756 views • 27 slides
Modeling Data the different views on Data Mining Views on Data Mining Fitting the data

Modeling Data the different views on Data Mining Views on Data Mining Fitting the data

Modeling Data the different views on Data Mining Views on Data Mining Fitting the data Density Estimation Learning being able to perform a task more accurately than before Prediction use the data to predict future data

517 views • 29 slides
Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction

Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction

Abstract Data Types & Templates 1 Abstract Data Type 2 Data and Abstraction Operations on data are central to most solutions Think abstractly about data and its management Typically need to Organize data Add data Remove

756 views • 30 slides
Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper

Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper

Tidy data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Tidy data Tidy Data paper by Hadley Wickham, PhD Formalize the way we describe the shape of data Gives us a goal when formatting our data Standard way to organize data

462 views • 27 slides
Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is

Big Data Analytics Armistead Boyd SVP, Product & Data Partnerships October 25, 2016 What is Big Data Analytics? Data Future Data Past Data Present 1 Big Data Trends Data Driven Organization Other Executives 5% 8% Operations Data

620 views • 22 slides
Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may

Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may

Concatenating data CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Combining data Data may not always come in 1 huge le 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have new data for each

555 views • 31 slides
Data Collection and HIVe Current Data Collection For those collecting data, you are use to

Data Collection and HIVe Current Data Collection For those collecting data, you are use to

Data Collection and HIVe Current Data Collection For those collecting data, you are use to the Excel data collection sheets All Ryan White Funded entities will be required to report client level data Non-Medical Case Management

266 views • 13 slides

More recommend