iRODS Consortium 2013 Reception 5:00 pm Welcome and Overview, - PowerPoint PPT Presentation

iRODS Consortium 2013 Reception

5:00 pm Welcome and Overview, Charles Schmitt 5:05 pm Introduction of new Executive Director, Brand Fortner 5:10 pm Update on iRODS Consortium, Reagan Moore 5:20 pm Technology Roadmap, Jason Coposky 5:40 pm Support Model – Expectations, Charles Schmitt 5:50 pm Features – Requests and Bug Process, Terrell Russell 6:00 pm Future Directions, Reagan Moore 6:10 pm Presentation, Use Case, Chris Smith, Distributed Bio 6:20 pm Presentation, Use Case, Pete Clapham, Sanger Institute 6:30 pm Presentation, Use Case, from Alan Hall, NCDC/NOAA 6:40 pm Wrap Up – Invitation to Thursday Tutorials, Charles Schmitt and Brand Fortner

Thank you Chris Smith, Distributed Bio Pete Clapham, Sanger Institute iRODS Consortium Executive Committee and Advisory Committee Members DICE & RENCI staff and management that have gotten us here 3

iRODS Users - examples Federal Users — ◦ National Aeronautics and Space Administration ◦ National Oceanic and Atmospheric Administration ◦ National Archives and Records Adminstration ◦ USGS Non-profit/Institutional Users — ◦ Broad Institute ◦ International Neuroinformatics Coordinating Facilities ◦ Wellcome Trust Sanger Institute Commercial Users — ◦ DOW Chemical ◦ Bejing Genome Institute Resellers/Redeployers — ◦ Distributed Bio ◦ Computer Sciences Corporation (CSC) Academic Users — ◦ Too many too list 4

iRODS Users - examples — Proven at scale: ◦ iPlant - 10k users ◦ French National Institute for Nuclear Physics and Plasma Physics – over 6 PB ◦ Wellcome Trust Sanger Institute – over 8 Pb ◦ Australian Research Collaboration Service - 512 storage resources ◦ NASA Center for Climate Simulations - 300 million attributes ◦ Cinegrid – sites across Japan-US-Europe — Solid foundation: ◦ SRB: initial product (developed by DICE Group, owned by General Atomics) in 1997 ◦ iRODS: rewrite of SRB by DICE Group in 2006; currently on version 3.3 ◦ Enterprise iRODS: mission critical distribution co-developed by RENCI and DICE in 2012 — Support: ◦ Community of developers from groups worldwide ◦ Independent groups offering consulting and support and development ◦ iRODS Consortium offering formal support, training, involvement, and development help 5

iRODS Users - downloads — As open-source software, we cannot fully track usage, only downloads: — 3.0 and 3.1 release: ◦ 1004 unique IP addresses combined (currently don’t have independent counts) — 3.2 release: ◦ 1202 unique IP addresses ◦ About 15% are confirmed commercial sites ◦ Another 15% are non-profit non-universities 6

iRODS Consortium — Provide: ◦ Open source iRODS release ◦ Single code source ◦ Sustainable development ◦ Binary distribution — iRODS data grid ◦ Organize data, information, knowledge ◦ Enable collaboration, publication, preservation ◦ Provide interoperability mechanisms ◦ Pluggable architecture for flexibility

Current iRODS Development — DICE Center — RENCI ◦ Features ◦ Pluggable architecture ◦ Sheau-Yen Chen ◦ Jason Coposky ◦ Mike Conway ◦ Zoey Greer ◦ Reagan Moore ◦ Harry Johnson ◦ Arcot Rajasekar ◦ Terrell Russell ◦ Wayne Schroeder ◦ Antoine de Torcy ◦ Hao Xu Parallel Software Development

iRODS 4.0 (March 2014) — Combine features from ◦ iRODS 3.3 (July 17, 2013) – Hadoop Distributed File System driver – PAM/LDAP authentication – Workflow structured objects – Rule language extensions – Jargon Java I/O library – netCDF and OpeNDAP support ◦ E-iRODS 3.0.1 (October 31, 2013) – Pluggable security (authentication, identity) – Pluggable micro-services – Pluggable storage drivers (hierarchical resources) – Pluggable network drivers – TLS security – Binary distribution – one-click installation

iRODS 4.0 — Hardened and tested code — One click installation (27 seconds) — Pluggable architecture — Federated security

iRODS Consortium ◦ Harry Johnson — iRODS Consortium ◦ Terrell Russell ◦ Brand Fortner ◦ Antoine de Torcy ◦ Charles Schmitt ◦ Leesa Brieger ◦ Arcot Rajasekar ◦ Michael Shoffner* ◦ Reagan Moore ◦ Lisa Stillwell* ◦ Wayne Schroeder ◦ Jason Coposky Joint development of combined release, iRODS version 4.0 by March 15, 2014

DataNet Federation Consortium — Migrating to iRODS 4.0 Features: – Spring 2014 Workflow registration Workflow re-execution ◦ Reagan Moore Workflow provenance OpeNDAP protocol ◦ Arcot Rajasekar NetCDF manipulation ◦ Wayne Schroeder Time series archiving Soft links to external repositories ◦ Sheau-Yen Chen Mediawiki integration ◦ Hao Xu VIVO integration AMQP messaging integration ◦ Mike Conway HIVE linked data ◦ Lisa Stillwell idrop-Web interface In-Common logon ◦ Charles Schmitt Dataverse integration Antelope integration ◦ Michael Shoffner Openflow controller integration

iRODS Consortium Members — RENCI — DICE Center — Max Planck Society — Data Direct Networks

Consortium Activities — Generate a development roadmap — Prioritize development tasks — Standard iRODS release — Service level support — Matchmaking between support providers and iRODS users ◦ Tier 1 ◦ Tier 2 ◦ Tier 3

iRODS Consortium Roadmap Technical iRODS Consortium iRODS Community 03/12 07/13 03/14 10/12 3.2 4.0+ 3.0 3.1 3.3 4.0 09/11 Plus, independent 3.0 3.0.1 plugin iRODS Enterprise 03/13 11/13 releases iRODS Consortium Development Team - Supercomputing 2013

State of the Union: E-iRODS 3.0.1 Features — Feature compatible with 3.0 Community, including bug fixes up to 3.3 — Up to date rule engine — Full SSL support with parallel transfer — Live hierarchy manipulation—move and rename resources in a hierarchy with existing data on the resources — Added Rebalancing as an operation—currently implemented for replication resource iRODS Consortium Development Team - Supercomputing 2013

State of the Union: E-iRODS Continuous Testing — Code coverage at 54% for the Agent, up from 28% originally — Plugin coverage on average at 70%, up to 93% — Continuously tested across 6 platforms — Full feature testing against 13 individual Resource Hierarchies — Topological Testing — Federation Testing, including 3.3 iRODS Consortium Development Team - Supercomputing 2013

State of the Union: E-iRODS Plugin Interfaces Current (3.0.1): ◦ Microservices ◦ Resources ◦ Network ◦ Authentication Coming Soon (4.0): ◦ Database iRODS Consortium Development Team - Supercomputing 2013

State of the Union: Current Plugins — Resource plugins: Replication, Round Robin, Random, Passthru , Compound, UnivMSS, MockArchive , UFS, Non-blocking, and Structured File Object — Network plugins: TCP and SSL — Authentication plugins: PAM, OSAuth, and Native iRODS Consortium Development Team - Supercomputing 2013

iRODS 4.0 ( March 2014 ) — Full merge between E-iRODS and up-to-date Community code — Repository migration to GitHub—transparent development process — Tickets, Workflows, approximately 50 updates not already included (e.g., filesystem metadata collection, storage admin role, sha1 ) — Comprehensive feature testing during merge Architectural Updates: — Database plugins—Postgres, MySQL, Oracle iRODS Consortium Development Team - Supercomputing 2013

Independently Released Plugins — Released as separate packages — Authentication (13Q4) ◦ Kerberos ◦ GSI — Resource (14Q1) ◦ S3 ◦ HDFS ◦ Load Balancing ◦ WOS (streaming)* ◦ HPSS 7.3 and 7.4 (expanded platforms)* iRODS Consortium Development Team - Supercomputing 2013

E-iRODS Plugins — E-iRODS 3.0.1 — Independently Packaged Releases Resource plugins: — — Resource Plugins: ◦ Replication ◦ Round Robin ◦ S3 ◦ Random ◦ Compound ◦ HDFS ◦ UnivMSS ◦ Unix File System ◦ HPSS 7.3 and 7.4 ◦ Non-blocking Network plugins: — ◦ WOS ◦ TCP ◦ SSL — Authentication: Authentication plugins: — ◦ PAM ◦ ◦ Kerberos OSAuth ◦ Native ◦ GSI

State of the Union: E-iRODS Plugin Interfaces — Reliability – Creates more easily tested software — Flexibility – Convert compile time options into run-time configuration — Stability – Create a hardened core and externalize rapid development — Community – open the development up to a large audience E-iRODS 3.0.1: iRODS 4.0: Microservices Database Resources Network Authentication

iRODS Development Areas — RESTful Interface—use the consortium as a vehicle for standardization — Refactoring—continue migrating toward more abstract interfaces ( e.g., special collections ) — Dynamic API—extend iRODS API as any other plugin interface — Plugin Registry—keep track of plugins and state in the catalog — Plugin dependency model—describe interdependencies between plugins, ship packages of plugins together as a collective feature iRODS Consortium Development Team - Supercomputing 2013