Data Systems Modernization (DSM) Project: Development, - PowerPoint PPT Presentation

Data Systems Modernization (DSM) Project: Development, Deployment, and Direction Robert Whitten Jr.

OLCF/NCCS Computing Complex #2 ¡ Peak performance 2.33 PF/s Memory 300 TB Disk bandwidth > 240 GB/s Square feet 5,000 Dept. ¡of ¡Energy’s ¡ Power 7 MW Jaguar ¡ most ¡powerful ¡computer ¡ Peak performance 1.03 PF/s #8 ¡ Memory 132 TB Disk bandwidth > 50 GB/s Square feet 2,300 Na7onal ¡Science ¡ ¡ Founda7on’s ¡most ¡ ¡ Kraken ¡ Power 3 MW powerful ¡computer ¡ Peak Performance 1.1 PF/s #32 ¡ Memory 248 TB Disk Bandwidth 104 GB/s Square feet 1,600 Na7onal ¡Oceanic ¡and ¡ ¡ Power 2.2 MW Atmospheric ¡Administra7on’s ¡ ¡ NOAA ¡Gaea ¡ most ¡powerful ¡computer ¡ 2

What is DSM? • Data Systems Modernization (DSM) • Software project to consolidate data sinks • Business intelligence tool • Data warehouse • Extract-transform-load (ETL) tool 3

What is DSM? (cont.) • Resource Allocation and Tracking System (RATS) – Projects, users, and allocations • NACS (New Account Creation System) – System accounts (usernames, file system areas, etc) • DowntimeDB – System status • HPSS stats – Archival usage 4

What is DSM? (cont.) Components • All middle-ware components used combination of: – MySQL Database – LDAP – Accessor / mutator scripts (Perl, Python, etc) • DSM adds: – ProcessMaker – LDAP Sync Script – Isolation Layer – System Sync Scripts (SSS) – Interface Scripts – LogiXML 5

RATS ¡ ¡ Cycle Servers ... Sch N Sch 0 o f I n b o J e v i c e e R / b o J y e r S u u b Q m i t J o b ??? Jobs Monitor Metascheduler Check Scheduled Job Info/Remove Info Stats from Consumption Consumption b Resource J o t e d m i t Report S u b Test Job Validity Report Job Charges Scheduled Jobs Scheduled Job Statistics Job Status Manager Jobs Dataset Job Status Admissibility Job Statistics Dataset Tester Dataset Job ID Registration Check Job Validity/Ack Resource Update Resources Charges Resource Char Valid Dataset ges ate Jobs ID Log Manager Check Machine Availabilty Host Projects Resource Status Jobs ID Configuration Projects Log Dataset Host Conf e RATS Validat Users Dataset Resource Status Dataset RATS Users RATS Users Static Static Attributes Attributes Dataset Platform Validate Users Dataset Platform Users Platform Users Dataset 6

NACS ¡ ¡ NACS ¡Scripts ¡ LDAP ¡ Data ¡Source ¡ NACS ¡Database ¡ Lustre ¡ NFS ¡ 7

DowntimeDB • Manual entry of downtime information Data ¡Source ¡ Reports ¡ Down7me ¡ Database ¡ 8

HPSS Stats • Data read directly from HPSS metadata Reports ¡ HPSS ¡ 9

Why DSM? • Multiple middle-ware applications used – To manage allocations (RATS) • Projects, Users / PIs, CPU Hours – To manage user system accounts (NACS) – To track downtime information (DowntimeDB) – To track storage usage (HPSS) • Redundant data • Inconsistent interfaces • Difficult report generation 10

DSM • Combine best features, remove inconsistencies DSM_NACS ¡ SSS ¡Views ¡ Interface ¡Scripts ¡ DSM ¡Database ¡ DSM_RATS ¡ ProcessMaker ¡ Report ¡Views ¡ LogiXML ¡ 11

ProcessMaker • ProcessMaker is open source workflow software solution – Business process management tool • Initially using it for account/project creation 12

Interface Scripts • Developed at ORNL to allow staff to modify user, group, project, etc. attributes – Add/remove user – Add/remove user from project – Create project • Written in python • Plan to migrate to ProcessMaker 13

LogiXML • Business Intelligence Tool • Management reports made easy? 14

When? • Phase 1 – Deploy on NOAA systems – No LogiXML – No ProcessMaker – Remote LDAP synchronization – Completed FY11 Q1 • Phase 2 – Deploy on DOE systems – LogiXML – ProcessMaker – Target FY11 Q4 15

Future Plans • Phase 3 – Expand role of ProcessMaker • Added functionality beyond account creation – RATS has an open source descendent • DataMux (available on Source Forge) • Replace the current isolation layer with DataMux components – Consolidate NOAA and DOE instances of DSM 16

Questions? 17