SAM Data Management Services Adam Lyon SAMGrid Project Leader - PowerPoint PPT Presentation

SAM Data Management Services Adam Lyon SAMGrid Project Leader – CD/REX/PS Leader – DØ SAM is a multi-level system for data management in use by DØ, CDF and MINOS Adam Lyon / Fermilab CD DØ / Neutrino Computing Workshop 1

SAM as a Data Catalog (all) Store metadata about files File type, run information, stream names, MC info, … Create datasets (lists of files) based on metadata queries Datasets are “live”, query language simpler than SQL Replica Catalog Maintain list of file locations (including pnfs & SRM locations) 2

SAM as a data “deliverer” (all, especially CDF, MINOS) Deliver file URLs upon request (interop with dCache, bluearc & SRM) Throttle deliveries to protect underlying cache/storage systems Move files from storage to cache and cache to cache world wide Track file usage by projects and jobs. Easy creation of recovery jobs 3

SAM as a cache management system (DØ) Operate a system of multi-tiered caches (large cache nodes, small cache nodes) Distributed cache for Grid jobs worldwide Complex routing possible 4

SAM has a “request system” (DØ for MC) Store metadata for a MC production request Track produced files (real and virtual) and apply metadata Easy queries for MC files 5

SAM is maintainable and scalable Tiny operational load at MINOS ~ 0.75 FTE for entire DØ data management system ~ 1.5 FTE for CDF data management system (but also includes dCache monitoring and operations) Low ongoing development (but still improving caching tune, keeping system up to date with latest infrastructure) SAM works with SL 4 and 5 SAM is easily scalable (including database access via DB servers) 6

!""# ,-.-/012341510/.6/756819.: !"" $&" $*" $(" $!" $"" ?@ &" *" (" !" " "$%"#%"# "$%$$%"# "$%$'%"# "$%$)%"# "$%$+%"# "$%$#%"# "$%!$%"# "$%!'%"# "$%!)%"# "$%!+%"# "$%"&%"# "$%$"%"# "$%$!%"# "$%$(%"# "$%$*%"# "$%$&%"# "$%!"%"# "$%!!%"# "$%!(%"# "$%!*%"# "$%!&%"# ;<-2=9->:54$ ;<-2=9->:54! ! ! 7

SAM has client and developer features Command line, Python, C++ Interfaces Interface SAM with your framework 8

SAM - you can take what you need Data catalog Data delivery (real or virtual) Data caching 9

Items to consider How much of SAM do you need to satisfy your requirements? Can your needs be consolidated? e.g. Can multiple experiments share a database? Share a SAM installation? Share a cacheing system? How much development is required (by you and us)? 10