Computing Report Construction Project activities Computing Consortium activities Capacity provision (GridPP+IRIS) On behalf of Manchester, RAL, Edinburgh (paid partners) All other participating GridPP sites (unpaid partners) Pete Clarke / DUNE-UK / DL / 11th Dec 2019 1
Specific construction Project activities 2
Construction project “paid” work Data Management Lead: Edinburgh • People: James Perry, Teng Li • • Starts: 2019 @ 1 FTE Immediate work: Mainly development of RUCIO data management for DUNE • • Share with RUCIO work for other communities in IRIS Offline Production Management Lead: RAL • People: Chris Brew, Raja Nandakumar • • Starts: 2021 @ 0.5 FTE but this is started anyway Initial Work: Trialling use of DIRAC (the LHCb workload management system) • Cloud Integration Lead: Manchester • People: Andrew McNab • Starts: 2020 @ 0.25FTE but this is started anyway • Initial Work: Augmenting DUNE capability to use diverse resources • 3
Data management @ Edi : Rucio Perry, Li Rucio is the ATLAS data management system • Policy driven • It is now adopted by CMS, DUNE • Being trialed by SKA and possibly LSST Work carried out (Perry) • Added support for S3 and Swift signed URLs to Rucio core • Added objectstore to Rucio development Docker image • Move current experiment-specific code out of Rucio core and into separate Python packages – Each VO will maintain its own policy package • Basic implementation done, targeted for Rucio 1.22 release Work carried out (Li) Monitoring including data deletion/movement, RUCIO internal healthiness and accounting . • 4
Data management: Rucio dashboard 5
Workload Management @ RAL : DIRAC Nandakumar, Brew Demonstrated usability of DIRAC as a WMS for DUNE è Run simple user jobs https://indico.fnal.gov/event/21328/session/0/contribution/7/material/slides/0.pdf l è Full chain MC test “production” https://indico.fnal.gov/event/21506/session/0/contribution/8/material/slides/0.pdf l è Integrated with SAM http://samweb.fnal.gov:8480/station_monitor/dune/stations/dune/projects/test_nraja-5770 l https://docs.dunescience.org/cgi- l bin/sso/RetrieveFile?docid=12982&filename=DUNE_MonthlyReport_201909.pdf&version=10 This work was commissioned by DUNE computing Presented at summer workshop Gives DUNE choices 6
Cloud resource work @ Manchester McNab 7
Computing Consortium 8
9 3
Andrew McNab is one of two technical coordinators This work not foreseen in the construction proposal • …..this is the story of computing • AMc made particular effort this year for Data model workshop in BNL in August • Organised and led computing model workshop in FNAL in Sept • PC setting up the Computing Contributions Board Formal body to deliver computing capacity requirements • • Follows LHC-like process One national representative from each of the larger partner countries + FNAL + CERN • Representation of DUNE in various relevant places important to DUNE. WLCG Management board (PC, AM) • LHCOPN Network meetings (PC) • • WLCG GDB steering group (AM) GridPP PMB (PC, AM) • • IRIS PMB (PC-Director, AM) 10 10
Computing capacity contributions 11
Computing capacity via GridPP&IRIS Capacity provided on Grid via GridPP DUNE uses Grid resources like any other large HEP experiment • Part of WLCG + OSG • Includes FNAL and CERN of course • October 2019: - 42 countries - 65 MoU’s - 168 sites - ~ 900,000 cores - ~ 0.5 Exabytes disk - ~ 0.5 Exabytes tape 12
Computing capacity from UK GridPP sites jumped in (resources and people to help) RAL -> R.Nandakumar (PPD) attends sites meetings • Manchester • Edinburgh • Imperial -> D.Bauer attends sites meetings - also IRIS scrutineer work • Lancaster -> M.Doidge active in enabling DUNE • Liverpool -> S.Jones attends sites meetings • Sheffield • Glasgow • • Oxford -> V.Davada attends sites meetings Bristol • QMUL -> T.Froy setting up perfSONAR • IRIS & GridPP have provided the extra resources GridPP5 has 10% for non-LHC • GridPP6 includes DUNE • • DUNE also submits formal annual request to IRIS (www.iris.ac.uk) In 2019 : 1000 cores + 2.0 PB of disk • In 2020 will be ~ 2000 cores and 2.5 PB - driven by DUNE model, not UK • 13
Computing Contributions so far Shown to LBNC Central Ring: Percentage of successful production jobs over last year • Central ring shows countries • Outer ring shows sites This shows a good trend Country % prod. jobs USA 52 UK 18 CH 12 Resources provided by: NL 6 • OSG sites • WLCG sites ES 2 • FNAL CZ 6 • CERN (part of WLCG) FR 5 14
From EGI accounting portal 15 15
User analysis jobs : a problem Percentage of successful user analysis jobs • Central ring shows countries Last 6 months • Outer ring shows sites Need to encourage analysis jobs to migrate to the world • Education/culture change So please consider sending your jobs to non FNAL sites 16
How we should be building our electronics 17
Recommend
More recommend
