HL-LHC ECAL off detector architecture Nikitas Loukas, University of Notre Dame N.Loukas, Aug 30 2017 FNAL Technical Review - Barrel Calorimeter 1
HL-LHC Bio M.Sc. in Modern Electronic Technologies, University of Ioannina, Greece: 2009 CMS Collaboration member: Jan 2012 – present. Based at CERN: Mar 2014 – present. PhD at University of Ioannina (GR): 2017 Thesis title: CMS L1 Trigger Upgrade - The Barrel Muon Track Finder Thesis main achievements: Development of the Barrel Muon Track Finder (BMTF) firmware and installation of the system in CMS. [Conference:C15-09- 28, JINST11(2016):no.03,C03038]. Professional Electronics Engineer working for University of Notre Dame, based at CERN: Jan 2017 – present. FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 2
HL-LHC ECAL upgrade on phase 2 - Modivation Max Latency Pipeline (BXs) Max L1A rate Trigger Phase 2 12.5 μ s 500 750MHz After Trigger Phase 1 3.2 μ s 128 150 KHz Requires a significant enhancement in finer granularity in the calorimeter trigger. Instead of TPs per Tower it is planned to have TPs per crystal or small clusters. Requirement: BCAL-sci-engr-002 The new granularity will increase bandwidth and processing power . The Requirement: BCAL-sci-engr-007 build of TPs will be moved from the FE to BE . The calorimeter trigger must be able to accommodate the required latency . Requirement: BCAL-sci-engr-005 The higher rates produce higher pileup which will be mitigated by precision timing in the calorimeter as well as spike killing . Requirement: BCAL-sci-engr-006 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 3
HL-LHC ECAL Block Diagram of the legacy system Trigger path DAQ path TTC & ctrl path Separate trigger and data cards Reads out only areas of interest due to bandwidth limitation 5x5 crystal trigger primitives FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 4
HL-LHC Front-End Phase 2 Upgrade In legacy ECAL, the TPs are build in the trigger towers FENIX chip sums of 5 crystals into a strip, analyze the digitized signal and performs Trigger and Readout functions 0.8 Gb/s links are sending all data to the VME BE. The sampling rate is 40 MHz In ECAL phase 2, the TPs will be build in the Back- End FE will have lpGBT chips that will concentrate row ADC data and send them to BE via Versatile+ links In addition FE will distribute the clock and will control the VFEs via I2C The sampling will be done at 160 MHz, which will help the BE to reject signal spikes Link utilization: Upstream 4 links at 10.24Gb/s with FEC5 and downstream 1 at 2.56Gb/s FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 5
HL-LHC ECAL Phase 2 Upgrade Overview L1 Trigger 16 GB/s FE links 10.24 GB/s Powerful FPGA links DAQ V V V L V V Feature extraction 16 GB/s links V F F F F F Suppress isolated E E E R E E anomalous deposits Trigger primitive 2.56 GB/s formation (signal fit, links TCDS + clock clustering, etc.) Motherboard HV Move trigger primitive generation to OFF-detector electronics S Requirement: BCAL-sci-engr-002 (FPGA) Three main tasks: Concentrate detector row data, build trigger primitives and transmit them to L1 Trigger. (402.03.03.04) Receive the LHC clock and distribute with high precision to the on detector electronics. (402.03.03.05) Form and send a event of data to the DAQ after each L1A signal arrived though the TTC interface. (402.03.03.03) FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 6
HL-LHC Clock and Control System Requirement: BCAL-sci-engr-006 Slide taken from: https://indico.cern.ch/event/649204/contributions/2639554/attachments/1483850/2302504/CMS_HL- LHC_EM_Calorimeter_Upgrade_20170628.pdf High Precision (HP) clock of 30ps is required for ECAL in the phase 2 upgrade. As baseline this clock will be delivered to the Front-End (FE) by the lpGBT downstream links. A fallback option is to deliver the HP clock directly through additional fibers. FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 7
HL-LHC Data Concentrator and DAQ links Waiting for L1A from TCDS++, build evens and send to the DAQ DTH provides two different data rates 16Gb/s and 25Gb/s 4x16Gb/s bidirectional links per FPGA. Requirement: BCAL-engr-039 8x16Gb/s bidirectional links per BCP ~120 Gb/s per BCP 8x16Gb/s 8x16Gb/s 8x16Gb/s … … B B B B B B B B B B B B D C C C C C C C C C C C T C - η + - η + P P P P P P P P P P P P H … … ATCA backplane FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 8
HL-LHC I/O link protocols The system will communicate with interfaces in different protocols and rates. Link rate Encoding bits/ Effective Comments BX line rate #1 800Mb/s 8b10b 16 640Mb/s ECAL shared links high rate #2 2.4Gb/s GBT 44 1.76Gb/s HCAL control and status #3 2.56Gb/s lpGBT 32 1.28Gb/s ECAL FE control and clock distribution – downstream #4 3.2Gb/s 8b10b 64 2.56Gb/s TCDS++ interface and LHC clock #5 5.0Gb/s 8b10b 96 3.84Gb/s HCAL data concentration – upstream BCAL-engr-004 #6 10.24Gb/s lpGBT 224 8.96Gb/s ECAL data Concentration – upstream BCAL-engr-004 #7 14Gb/s 64b66b 320 12.8Gb/s ECAL shared links high rate, interface to DAQ, interface to Calo Layer-1 #8 16Gb/s 64b66b 384 15.36Gb/s FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 9
HL-LHC FPGA baseline The curves show No Family FPGA LUT (K) DSP BRAM (Mb) GTH GTY MGTs resources of Ultrascale 0 Virtex-7 XC7VX690 433.200 3,600 51.68 80 - 80 FPGAs (that meat the 1 Kintex Us XCKU085 497.520 4,100 56.9 56 - 56 transceiver number) 2 Kintex Us XCKU115 663.360 5,520 75.9 64 - 64 versus the resources of 3 Kintex Us+ XCKU15P 1,143.000 1,968 70.6 44 32 76 the reference 4 Virtex Us XCVU160 926.400 1,560 115.2 40 36 76 XC7VX690 used in CTP7. 5 Virtex Us XCVU190 1,074.240 1,800 132.9 40 36 76 6 Virtex Us+ XCVU5P 601.000 3,474 36 - 76 76 The Energy and timing 7 Virtex Us+ XCVU9P 1,182.000 6,840 75.9 - 76 76 information will be 3.000 extracted using DSPs with a minimum 2.500 requirement of 1800 LUT (K) 2.000 slices. DSP BRAM (Mb) 1.500 XCKU115 is the MGTs 1.000 baseline, capable to Reference FPGA accommodate both DSP threshold 0.500 ECAL and HCAL FPGA / 0.000 XC7VX690 XCKU085 XCKU115 XCKU15P XCVU160 XCVU190 XCVU5P XCVU9P Package B2104 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 10
HL-LHC Processing region per FPGA 15 ECAL Towers: 3 η x 4 φ 14 13 Region: 300 crystals 12 15 η x 2 0φ = 0.26 x 0.35 11 Tower crystal 10 216 UltraScale FPGAs 9 8 7 ATCA blades 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 11
HL-LHC Front-End to Back-End links 34 Total number of 33 15 32 31 FE to BE links: 12240 14 30 29 5 28 13 27 One Tower 26 4 12 25 Number of links 5 lpGBT links 24 11 23 Per BE card is 60 + 3 22 1 down 10 21 4 up 20 2 9 19 η 18 17 8 1 16 15 7 One FPGA - 12 Towers 14 - 13 6 1 2 3 4 5 12 11 5 10 9 4 crystal 8 7 Total number of Tower 3 6 5 EB Back-End FPGAs 216 4 2 3 2 1 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 φ Tower 20 degrees (2 Super Modules) 12 Towers x 5 lpGBT = 60 links (48 upstream & 12 downstream) Requirement: BCAL-engr-017, BCAL-engr-018 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 12
HL-LHC Shared Back-end links (in case we need) 3 3 Right or left 2 2 Swiss cross 1 1 neighbors: 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 30 Crystals x 16 14 14 13 13 bits x 40MHz = 12 12 11 11 19.2Gb/s => 10 10 9 9 2 links @16Gb/s Tower 8 8 7 7 6 6 5 5 12 Towers per 4 4 3 3 BE card 2 2 1 Cluster Δη x Δφ = 3 x5 xtls 1 Corner neighbors: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2 Crystals x 16 bits x 40MHz = 1.28Gb/s => Top or bottom: 20 2 links @0.8Gb/s Crystals x 16bits x 40MHz = 12.8Gb/s 8 Transceivers in total => 2 links @16Gb/s 4 Right/left 4 Top/Bottom 8 SERDES 2 on each corner Requirement: BCAL-engr-023, BCAL-engr-026, BCAL-engr-027 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 13
Recommend
More recommend
