CMS Data Transfer tests towards LHC data taking CMS Data Transfer tests towards LHC data taking D � Bonacorsi � CMS Facilities � Infrastructure Operations ��� INFN � CNAF � Bologna � Italy � On behalf of the CMS experiment *Thanks* to all site people and operators.
Main CMS work � ows � simpli � ed � Prompt reco reco Prompt Data Transfer Data Transfer Skims, re-reco re-reco Skims, Calibration Calibration Data Transfer Data Transfer Data Transfer Data Transfer MC prod upload Skims download MC prod upload Skims download In this talk: focus on distributed transfer only D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
PhEDEx Physics Experiment Data Export reliable � scalable data replication system for HEP experiments CMS is exercising the data placement system since ���� PhEDEx fully interfaced also with gLite FTS since yrs CMS data transfers in a nutshell: � T �� ����� T � � T �� ����� T �� � s � �� T � s � �� T �� � s � �� s � �� T � T �� � s s in current PhEDEx transfer topology ���� of CMS Institutes Data transfers operated as if experiment was already running we have had � service outages exceeding �� hrs in the last � yrs Current transfers at ��� ��� GB � s GB � s � � ��� TB � day ��� TB � day � � � PB � month � PB � month global average rate average CMS � les size likely to be �� GB � � � �� k � les � day �� k � les � day D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
Evolution of a LoadTest � from early ���� on � A � exible infrastructure to generate data transfer tra � c among CMS Tiers Start moving � fake � but � real � No real physics � les; fully PhEDEx � compliant � though ����� activity ����� activity since mid � February ���� full cycles ��� weeks each � before Jun �� � then extended into DDT � CSA �� preparation activities � T � � T �� tape �� T � � T �� T � � T � � regional �� T � � T � � non � regional � Jan 2007 Jan 2007 mid-March 2007 mid-March 2007 D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
Start walking Jan 2007 mid-March 2007 Jan 2007 mid-March 2007 Cycle-1 Cycle-2 Jan 2007 mid-April 2007 Jan 2007 mid-April 2007 D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
Walk better and faster Jan 2007 mid-April 2007 Jan 2007 mid-April 2007 Sep 2006 Today Sep 2006 Today D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
CMS CSA06 : CMS LoadTest LoadTest 2007 2007 CMS ~1 PB in ~1 month to participating Tiers >12 PB in ~6 months among all PhEDEx Tiers joining the LoadTest D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
[ courtesy of L.Tuura, CHEP07 ] 2007/Q12 D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
Data transfers: grandview Status � as of mid ������ : With � yr to LHC start � up � CMS approaches the real transfers in scale � but not yet the full complexity From reliable transfers over the full transfer mesh � to multi � VO exercises… The progress is evident � though � Main sources of this are: A well � designed � robust � scalable transfer system A remarkable manpower investment to commission the transfer system Continuing e � orts are needed on debugging data transfers debugging data transfers more D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �
[ courtesy of L.Tuura ] Screenshot as of Sep 2007 D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
Quality of transfers Clear improvement in Tiers participation to test transfers since LT started Still not evident improvement in quality � though It � s not one problem that lacks solution � there is a wide span of them Greenish quality plots � successful transfers with fewer retrials � only when storage at both ends � network � PhEDEx set � up � site con � g � operators work � ne… simultaneously! 30 20 # Tiers making some successful transfers # Tiers making transfers at >50% quality Jan06 Jun07 LoadTest LoadTest CSA06 CSA06 Positive note: now stably at a ~challenge traffic load Need to: Keep sites exercised + Debug and improve quality more D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
DDT Debugging Data Transfers A CMS program to maintain a high � quality transfer network to be handed over to CMS Data Operations Started in July 07 Debug � commission transfer links transfer links among CMS Tiers a Task Force � DDT � TF � is in charge since July ���� Joined e � ort with CMS Facilities � Commissioning � T � liasons � PhEDEx � Central Ops � FTS � SRM experts � network experts � site admins � … Troubleshooting � by Tiers � work by milestones focus on watching logs � ping site admins � � x problems commission T � � T �� T � downlinks to T �� T � uplinks to T �� … Gain confidence Infrastructural issues � by Facilities � Network projects Overall activity � DDT � TF � work on deliverables E � g � a real � time status map � with reasons � of all Tier � X � Tier � Y links E � g � a number of documented � success stories � in troubleshooting D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
DDT The � rst DDT metric First step was to de � ne and implement a metric by which links can become � commissioned � and subsequently handed over to Data Operations There are several stages through which a link can get commissioned: � � NOT � TESTED: links never actually tested NOT � TESTED i � e � showing no successful transfer attempts within PhEDEx � � PENDING � COMMISSIONING: links that have transferred successfully at PENDING � COMMISSIONING least � � les in PhEDEx � but have not yet passed the reqs below � � COMMISSIONED: links that are demonstrated to work � * �� and can be COMMISSIONED delivered to Data Ops � Transfer ��� GB � day for � out of � consecutive days � and transfer a total of ��� TB during that same �� day period � For links involving an endpoint at a T �� this req is relaxed to � out of � days � and a total transfer volume of ��� TB � to match service business hrs support at T �� s � � � PROBLEM � RATE: links that were working but whose rate has dropped o � PROBLEM � RATE � To remain COMMISSIONED � a link must transfer at least ��� GB � day for a single day at least once every � days � Otherwise � the link must be re � commissioned by following the procedure above � � * � i � e � this does not imply that the link or the site has met the reqs of the CMS Computing Model � but simply that the link has passed some passed minimum reqs to be considered usable for Data Operations � namely: production � quality activity � not tests � D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
DDT The DDT status in late ���� Steady in � ux of new links… number of COMMISSIONED links that were in danger of decommissioning within the next two days Legenda: T1 � T1 ROW � COLUMN: upper half of box T1 � T2 COLUMN � ROW: lower half of box States: [ … plus many more … ] D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
DDT Going beyond The requirements � thresholds in this metric: � were developed with the idea of having a higher threshold to commission than to decommission the link � can be increased in time as networks and sites develop the rates implied by ��� GB � day are of the order of ��� MB � s per link � far below the commitments envisioned in the computing model T �� s being able to download a total of � up to � TB � day from T � sites � or over �� MB � s sustained downloads � � match the idea that transfers would be at continuous rate over several days � The Computing Model actually envisions that transfers will occur in bursts the metric used during CSA �� deviated from this model to prove the stability of data transfer links It worthed a metric revision later � But � before � see what happened with this one metric! D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
Before DDT started (1 month, May 2007) The plots show the fraction successes/attempts in file transfers. A clear improvement in the number and quality of data transfer links is seen soon after DDT started (Jul 07). After DDT started (1 month, Oct 2007, during CSA07) D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
In the meantime…. CCRC’08 /phase-1 (WLCG C ommon-VO C omputing R eadiness C hallenge) D � Bonacorsi Bonacorsi ISGC Symposium � Taipei � ���� April ���� �� ��
Recommend
More recommend
