Dune computing Workshop Cédric Serfon Cedric.Serfon@cern.ch On behalf of the Rucio team
Rucio in a nutshell (1) ● Rucio is a Distributed Data Management system built initially for the ATLAS experiment ○ It allows to federate the data located on different heterogenous sites (small/big, grid/cloud, Tape/Disk) under a common namespace and hide the complexity of the underlying storage layer ○ It provides tools to manage efficiently the data according to the policies defined by the collaboration, optimize the space, etc. ○ It provides tools for the the end-users to interact with the data ● It was built using more than 10 years of experience in Data Management: ○ Designed from experience from the previous ATLAS data management system DQ2 ○ Integrate new features and technologies ● Modular, highly scalable, well supported 2018-10-08 Rucio - Dune Workshop
Rucio in a nutshell (2) ● Who is using Rucio ? ○ Used by ATLAS, AMS and Xenon1T . CMS just choose to move to Rucio for LHC Run3 ○ Being evaluated by other small/medium/big HEP/Astro experiments: ● 1st Rucio community workshop was held on March 1st-2nd 2018 to present Rucio to more collaboration/scientific communities ○ Attended by more than 80 people ○ A new workshop will be organized in spring (3 instituts already candidated to host the meeting) 2018-10-08 Rucio - Dune Workshop
Rucio main functionalities ● It provides many features (you are not forced to enable all) : More advanced features ○ File and dataset catalog (logical definition and replicas) ○ Transfers between sites and staging capabilities ○ User Interface and Command Line Interface to allow user to download/upload/transfer their data ○ Extensive monitoring ○ Powerful policy engines (rules and subscriptions) ○ Bad file identification and recovery ○ Dataset popularity based replication ○ … ● Rucio can be easily integrated with Workload and Workflow Management System ○ Already supporting PanDA (ATLAS WFMS) ○ Possibilities of integration with other like Dirac 2018-10-08 Rucio - Dune Workshop
Rucio in ATLAS ● Rucio stores all ATLAS data : ○ Data coming from the detector ○ Monte Carlo data ○ User data ● Rucio takes care of : ○ Ensuring the replication of files according to the replication policy specified by ATLAS ○ Replicate the data for other applications (e.g. panda) and for the end-users ○ Ensure file recovery ○ Staging data from TAPE ○ And plenty other things 2018-10-08 Rucio - Dune Workshop
Rucio in ATLAS ● ATLAS instance in a few numbers ○ More than 1B files, ~0.4 EB ○ Up to 4M files/2.5 PB transferred per day ○ More than 1000 active users ● Expect to gain one order of magnitude for Run4 2018-10-08 Rucio - Dune Workshop
Rucio main concepts - Accounts ● Each user who wants to use Rucio needs an account ● There are different types of accounts : user, service, group ● One user can have different accounts and can use different credentials (X509, kerberos token, userpass, ssh) to connect to Rucio ● Permissions and quotas are based on accounts 2018-10-08 Rucio - Dune Workshop
Rucio main concepts - DIDs ● The data stored in Rucio are identified by a Data IDentifier (DID) ● There are different types of DIDs : ○ Files ○ Datasets : collection of files ○ Container : collection of dataset and/or container ● ● Each DID is composed of : DIDs hold a set of basic metadata e.g. ○ ○ A scope : 25 characters to partition your data, e.g. Bytes ○ data17, mc17 Checksum (for files) ○ ○ A name (up to 255 character) Number of events ● ○ A name is unique within a scope Datatype 2018-10-08 Rucio - Dune Workshop
Rucio main concepts - RSEs ● Rucio Storage Elements (RSEs) are logical entities of space ○ No software needed to run at the site ○ RSE names are arbitrary (e.g., "CERN-PROD_DATADISK", "AWS_REGION_USEAST", … ) ○ Usually one RSE per site and storage data class ● RSEs collect all necessary metadata for a storage ○ protocols, hostnames, ports, prefixes, paths, implementations, … ○ data access priorities can be set (e.g., to prefer a protocol for LAN access) ● RSEs can be tagged ○ Key/Value pairs (e.g., country=UK , type=TAPE , support=brian@unl.edu ) ○ You can use RSE expressions to describe a list of RSEs (e.g. country=UK&type=TAPE) 2018-10-08 Rucio - Dune Workshop
Rucio main concepts - Rules and subscriptions ● Replication rules : ○ Describe how a Data IDentifier (DID) must be replicated on a list of Rucio Storage Elements (RSE) ○ e.g. : Make 2 replicas of dataset data15_13TeV:mydatasetname on tier=1&disk=1 ○ Rucio will create the minimum number of replicas to optimise storage space, minimise the number of transfers and automate data distribution ● Subscriptions : ○ Replication policies based on Data IDentifiers metadata, for Data IDentifiers that will be produced in the future ○ e.g. : Make 2 replicas of datasets with scope=data15_13TeV and datatype=AOD on tier=1&disk=1 2018-10-08 Rucio - Dune Workshop
Rucio future developments ● New functionalities keeps on being added. Most recent ones ○ Support for generic metadata added this summer ■ I.e. now Rucio can support whatever key:value pairs on DIDs. This was a requests of many collaborations during the 1st Rucio workshop ○ Support for archive files ■ This new feature allow to registers the constituents of an archive file ■ Rucio can extract automatically the constituent of the archive ● More to come soon : ○ New authentication/authorization based on Macaroon/SciTokens ○ Support of different QoS for the storages 2018-10-08 Rucio - Dune Workshop
Conclusion ● Rucio is an open source project, everybody can contribute. ○ Right now the core development team is composed from people from ATLAS and CMS ● All the members of the team also provide support ● Rucio is a Data Management solution that demonstrated that it can scale for large collaboration O(1 EB) ● We’re very glad that CMS recently choose Rucio as their next Data Management system and hope that more collaborations will follow this path ● As mentioned in previous talk, DUNE has already a Rucio prototype 2018-10-08 Rucio - Dune Workshop
More information Website http://rucio.cern.ch Documentation https://rucio.readthedocs.io Repository https://github.com/rucio/ Continuous Integration https://travis-ci.org/rucio/ Images https://hub.docker.com/r/rucio/ Online support https://rucio.slack.com/messages/#support/ Developer contact rucio-dev@cern.ch 2017-01-01 13
Recommend
More recommend
