FJPPL Computing Workshop Operational experience with second machine - PowerPoint PPT Presentation

Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules FJPPL Computing Workshop Operational experience with second machine room at CC-IN2P3 Xavier Canehan

Introduction  2 computing rooms at CC-IN2P3 since 2011  Critical choices upon conception lead to consequent advantages  Adaptability remains mandatory  Monitoring and testing even the building  Drawbacks 2 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules 2 computing rooms Square feets and high power consumption 3 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Vil-2 initial objectives +10 year perspective Hot water Modernity Modularity Ease of Multi-Tier deployment architecture 4 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

High Quality results of Initial Conception  Modern computing room (details shown during visit) Initial plan for 2011 - 2019  Multi-tier by design Target:  3 phase deployment 2011 2015 2019 ◦ first one dedicated to 50 racks 125 racks 240 racks 0.6 MW 1.5 MW 3.6 MW computing farm ◦ relying upon regular Tier II Tier III Tier III-IV budget 5 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

First phase: Tier II over 2 lines 28 InRow Cooling units,18 – 20 kW each  One 2 MVA UPS chain of 4 * 500 kVA UPS  2 transformers of 1600 kVA  3 chilling units for 2,4 MW, only one distribution circuit.  Backup through a 24m^3 water tank. 2 power lines: dedicated main up to 9 MW, 2 MW  TIER II reservation on backup line ⅓ floor space used at each level  TIER II 6 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Site mean power usage, PUE advantage to Vil-2 Site power er co consumpti sumption on 800 ar around nd 1. 1.1 1 MW MW 700 600 Mean Power wer Usage age IT agai ainst nst Total al Room kW IT kW Total PUE PUE 1 (kW kW) 500 kW IT Vil-1 320 720 (-130) 1.84 400 kW Total Vil-2 300 440 1.46 300 200 Best PUE UE in in Vil Vil-2 100 Movin ing from Vil il-1 to V Vil il-2 0 gain ins s ~20% of of power cost Vil-1 Vil-2 7 7 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Entry ticket cost vs fully functional Vil-1  PUE is not linear: works by step, intercept is not null ◦ Beside investment costs ◦ Operational cost of electrical infrastructure must be taken in account  eg 1 UPS consumes up to 3kW  Other costs of investment ◦ Water cooled racks value ◦ PDU and rack power protection costs  Vil-1 is fully redundant, deals with hygrometry  Vil-2 initial target was deliberately limited No p poin int t in in d dit itchi hing ng Vi Vil-1 8 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules Moving grounds Adaptation remains mandatory 9 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Have to evolve  Environment changes  Needs and perspectives clarify and evolve  IT technology is volatile  Infrastructure pace stays slower  Monitoring everything 10 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Environment – IT densification effects upon racks  Space floor is no more a problem kW/m² increase Power  14 C6200 Dell sockets # per rack PowerEdge will ◦ approach InRow Cooling IT densification unit capacity (18kW) ◦ need 45 sockets, 3 phases PDU ◦ (dedicated PDU development) At rack limits Partially filled racks to limit constraints 11 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Environment – Power costs evolution  5% to 7% of annual /annual power consumption power cost increase /annual power cost, without tax / kWh cost, in € cents  Looking for IT efficiency  Fine tuned power contract helps to minimize costs Se Seek the more effi ficien cient IT ha hardwa ware in the most effici ficient ent room 12 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Power efficiency, ASHRAE recommendations  Actual hardware bears 35 ° C all year long  Increasing room temperature lowers overall power consumption 25°C server -40% fan entry activity for 11 kW Gain setpoint cooling units  Less noise: 95-110 dB to 85-95 dB  Hot corridor temperature also increased Corridor temperature 36 36°C-40 40°C What will be next setpoint ? 13 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Planning to cope with scientific needs in foreseable future Estimated imated #Racks ks What if all new hardware goes into Vil-2 ? 140 120 Estimated imated IT power er [kW] W] 100 2500 80 2000 60 40 1500 20 1000 0 2014 2015 2016 2017 2018 2019 500 Estimations from LHC and LSST/Euclid/CTA figures • Data modified with current densification factor • 0 Need Vil-2 adaptation to host storage systems 14 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Initial Plans revision  Previous power and cooling distribution systems implied evolution by pair of hot corridors Infrastr astruc uctur ture e cost for each new la lane: ~1.5 .5 M €  Minimizing costs by actual infrastructure hardware reuse Adapt pting ing exis istin ting g le least used ed Tie ier II II la lane 2 p phase ase pla lan le less s than 350 k k € per year  2017 aim: ◦ 80 racks, 1 MW IT ◦ 1 lane Tier II and 1 lane Tier III 15 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Introducing Tier III – Phase 1, power redundancy 2015 15 TIER III Hot Aisle C/D Used exten tension on 2015 2015 Used Hot Aisle A/B TIER II ext. details to be seen during visit 16 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules Drawbacks and limits From details to major drawbacks 17 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Infrastructure limits are easily forgotten  Coping with interdependant limits ◦ Cooling capacity or cooling redundancy ◦ Power capacity and Power redundancy ◦ Per rack, group of racks, aisle, distribution line  Multi-tier ability adds an order of complexity ◦ Event more if you mix tiers in a single line  Strict deployment plans needed Monit itoring ring is is manda dato tory 18 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Dealing with cooling defect Stopping InRow cooling units for 20 minutes → +15 ° C Increases corridor temperature around 49 ° C Increases front temperature to 43 ° C Need an effi ficient ient shut utdow down system tem Our water tank provide ides 20 m min in dela lay 19 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Smart versus dumb shutdown systems  Smart shutdown relies upon IPMI ◦ detects low continuous slope ◦ or fast temperature change  IPMI needs network  If network switches shut down before servers, IPMI is useless  Need a dumb backup power cut system Won’t reproduce a bad experiment with a water leak -20 ° C on roof, +65 ° C in hot corridor 20 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Cooling technology: hot water choice  Improving global Campus Energy Reuse Efficiency ? • Agreement to provide hot water on campus Land nd procu ocureme rement nt • Very efficent chillers, allowing reuse of hot water • Silent hardware Hot t water ter need ed • Campus is late • 3 years spent Fixed xed tech chnolo ology 21 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

No humans: no window, no faucet ? 22 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Cooling technology: improving efficiency  Won’t ever be able to use direct Free Cooling  But new cooling technologies are available New cooling Change IT technologies procurement 23 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Very valuable modularity outcomes  Ceiling rails  Preset pipes  Movable separation wall between used and free space  Roof as technical level 24 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015

Questions? Thank you! 25 FJPPL Computing Workshop – Operational Experience – Xavier Canehan 10/03/2015