dCache @ Fermilab Dmitry Litvintsev Dmitry Litvintsev, 6th - PowerPoint PPT Presentation

dCache @ Fermilab Dmitry Litvintsev Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Science at Fermilab Fermilab: Home of the Tevatron and buffalos • Energy frontier: BSM, precision EWK, Higgs, HF, QCD • CDF, D0 (ramping down), CMS T1 • Intensity Frontier: BSM, neutrino physics, rare decays • Astrophysics: dark matter search, high energy cosmic rays, galaxies surveys Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

dCache @ Fermilab total online on tape version public 0.1 PiB 2.5 PiB 1.9.5-28 Size CDF 1.5 PiB 10 PiB 1.9.5-22 CMS 13.5 PiB 17 PiB 1.9.5-23 httpdDoors: http://cdfdca.fnal.gov:2288/ http://fndca.fnal.gov:2288/ http://cmsdcam.fnal.gov:2288/ Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Tape Backend • 7 x SL8500 tapr robot libraries managed by Enstore HSM • http://www-ccf.fnal.gov/enstore/ • LTO4 and T10K tape drives • Total data on tape - 38 PB Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Tape Backend • 7 x SL8500 tapr robot libraries managed by Enstore HSM • http://www-ccf.fnal.gov/enstore/ • LTO4 and T10K tape drives • Total data on tape - 38 PB Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

public dCache PnfsManager movers w-pools 4x dcap 2x Kdcap PoolManager movers r-pools 2x gsi dcap SRM httpd 2x K FTP rw-pools admin 7xSL8500 movers 14x GFTP Enstore v-pools gPlazma1 dCache XACML WFTP HSM GUMS Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Pool Setup • Experiments buy pool hardware • Their data directed to their pools by regex match between ugroup store units and file storageclass extracted from layer 4 (storage_group + file_family combination + HSM name) psu set regex on psu create unit -store ktev.ktev@enstore psu create unit -store des.*@enstore ... psu create ugroup DESSelGrp psu addto ugroup DESSelGrp des.*@enstore psu create ugroup KTeVReadSelGrp psu addto ugroup KTeVReadSelGrp ktev.ktev@enstore ... Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

transfer rates @ public dCache Writes Bytes Written 1.5e+13 Bytes / day dCache Enstore 10 TB/day 1e+13 5e+12 Bytes 0 -5e+12 -1e+13 -1.5e+13 11-03-01 11-05-01 11-07-01 11-09-01 11-11-01 12-01-01 12-03-01 12-05-0 Reads Date (Year-month-day) Sun Apr 15 01:47:31 2012 Bytes Read 2.5e+13 dCache Enstore 2e+13 Bytes / day 1.5e+13 10 TB/day 1e+13 Bytes 5e+12 0 -5e+12 -1e+13 -1.5e+13 11-03-01 11-05-01 11-07-01 11-09-01 11-11-01 12-01-01 12-03-01 12-05-0 Date (Year-month-day) Sun Apr 15 01:47:25 2012 Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

transfers by experiment @ public dCache minos 22% lqcd cdms 21% 24% e907 des e906 theory 1% 1% 0% 0% exp-db 32% theory e906 des e907 lqcd minos cdms exp-db Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

transfers by country US 85% DE 6% NL 2% ES 2% IT FR 2% PT BR 1% CZ CO MX CH SI GB AU JP 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% AU GB JP MX CH SI CO CZ BR PT FR IT ES NL DE US Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

pnfs->chimera • required to develop chimera aware Enstore client (encp v3_10e) • to support direct encp required customized DB triggers to fill location information from layer1 and layer4 and enstore2chimera migration stored procedure. See https://srm.fnal.gov/twiki/bin/view/DcacheCorner/ChimeraEnstore • converted pnfs -> chimera on Feb 23rd, 2012 • 14718667 files pnfsDump 6h52m SQL import 9h47m enstore2chimera 1h8m import of companion 17m md5sum verification 113h24m Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Reliability • All planned downtimes included, public dCache uptime is 99.2% Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

plans • migrate pnfs -> chimera @ CDF and D0 (May 1st) • upgrade public dCache and CDF dCache to 1.9.12-x • provide nfsv4.1 server for IF experiments @ public dCache Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

public dCache for IF • Intensity Frontier (IF) experiment require POSIX I/O • => use dCache as backup storage. Use srmcp to copy data to BlueArc for access via NFS. • Concerns of long term scalability of this solution • We offered test dCache install with nfsV4.1 server for IF use case studies. Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

nfsV4.1 for IF VM Client VM Client dCache Pool Node (dmsdca09) dCache Pool Node (dmsdca10) /dev/sdb1 /dev/sdb2 /dev/sdb1 /dev/sdb2 2Gb FC 2Gb FC Fermicloud Nexsan SATABlade Nexsan SATABlade 1GigE Test dCache pNFS/nfsv41 - enabled Linux clients exports nfsv41 System filesystem dCache Head Node (dmsdca08) VM Client VM Client Client side: Server side: • SLF6 VM @ Fermicloud • dCache 1.9.12. 1 head node, 2 pool nodes. • 2.6.40 kernel • pool nodes - 2 RAID6 w/ 4x250GB SATA drives Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

Iozone in cluster mode with sequential r/w tests 300 3 00 MB/sec MB /sec 25 0 250 200 200 15 0 150 100 100 IOzone test Sequential write 5 0 IOzone connectivity test 50 Sequential read Sequential read 0 0 0 2 4 6 8 10 0 20 40 60 80 100 120 140 160 180 200 N umber of c l ients Number of clients One 4GB file per client (twice the VM Multiple clients per VM memory), one client per VM Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

mdtest 300 3500 Operations/sec Operations/sec 3000 250 2500 200 2000 150 1500 dCache mdtest ops 100 Dir creation 1000 Dir removal Tree creation dCache mdtest ops File creation Dir stat File removal 500 50 Tree creation File stat Tree removal 0 0 2 4 6 8 10 12 14 16 18 20 0 0 2 4 6 8 10 12 14 16 18 20 Number of clients Number of clients mdtest w/ multiple MPI tasks create/stat/remove 100 directories w/ 100 zero length files (stats performed with read-your-neighbor option to avoid reading locally cached files) Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012

nfsV4.1 for IF • nfsV4.1 performance is adequate to meet IF experiments’ requirements • further tests with real data and experiments workflows are foreseen Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012