Comments on DUNE DAQ Challenges Architecture Ba Babak Abi DUNE DAQ Simulations Meeting 16 16 Oct 20 2017 17 Review : • Alternative DAQ Architectures • DUNE DAQ data fellow • More about GPU based Architectures 1
DAQ Architecture; comments on note 4481 • Comments and discussion on note 4481: • SLAC ( Ryan&Matt,..) , Rick, Dave . • Components Life time • Modularity and scalability • Noise and detector performance • Optical & network Links , data error check • Reserve DAQ capacity ( in case of bad detector performance ) • Using the commercial components • ….. 2
DUNE DAQ Architectures • We (DUNE DAQ ) already have at least 3 1. RCE 4th generation upgrade for DUNE SLAC and Oxford work on 4 th generation DPM right now. 1. 2. GPU based 1. This talk and next talk from Phil. 3. FPGA based (simplified RCE?) (detailed talk next week) 4. Other Alternatives? (and DaqD3 and 4 in note 4481) • Major questions : 1. Ring Buffer 2. Trigger generation factors (Detector performance & algorithms ) 3. Time and clock accuracy (1uS tolerance time accuracy! & No need to phase locked Clock to each APA!) 4. Dual-phase merge& compatibility 5. What more ? 3
DUNE DAQ data fellow • Time/Clock/ Trigger Master -control distribution Timing, trigger,.. 2 2 APA DAQ 1 Flange 2560 channels Main DATA FPGA + CPU 100GE stream Network A 1 Global 3 Secondary Network Control/monitor/Data 4 Network B Trigger Processor Farm 40 SiPMs SSP 10 Photon Detection System per APA 4
Data fell llow; ; Rin ing Buffer & Tri rigg gger Pri rimitives Trigger Processor SSP Global Trigger Ring Farm Photon Trigger Primitive Beam,Random,Calibration Timing tag Buffer SN,Proton,Cosmics Detector Generator 1- Lower bandwidth for triggered DATA 2- Relaxed Trigger latency -> send through TCP packets back to buffer holder Board-Reader Triggered DATA 3- Global Time Event-Builder BACK-END DAQ 4- Separate clock -event builder can handle triggered data ? TPC TPC Ring Trigger Primitive Front-end Timing tag Buffer Generator Electronics 5
DaqD1 : Min inimum In In-detector architecture Detector side Off-Detector Global Time side Global PTP network 50MHz Clock Trigger CAT6 at Surface CAT6 distribution APA Processor unit 2560 channels FPGA/ASIC 10x10Gb/S PCI +GPU 100Gb MUX 80 x 1.25Gb serial 10 x 10Gb 10x10Gb/S Buffer + Links serial Links Tagging + compression 20:1 MUX CWDM 1x fiber pairs APA 2Km to surface 2560 channels FPGA/ASIC GPU 100Gb MUX 10x10Gb/S 80 x 1.25Gb serial 10 x 10Gb Links serial Links Processor unit Network PCI + GPU Buffer + Tagging + compression Another alternatives like : It is possible to send DATA through TCP/UDP GPU Packets through optical fibre but constrains on NIC and FPGA! PC- backend DAQ 6
DaqD1 : : Min inimum In In-detector architecture Off-Detector Global Time side Global PTP network Trigger CAT6 at Surface 1 APA requires : CAT6 1 Sever, 1 PCI card, 2/4 GPU, 2x10GE NIC 200GB DDR4 RAM as Ring Buffer Global Time • PCI card Time PTP PTP network • PCI 5 ready in 2019, 63 GB/s(x16) CAT6 • Commercially availed BUT an RCE with PCI extension can do the job! • Ring Minimum Firmware R&D • Buff Ring Buffer time stamp: PTP Time er • All CPU/NIC clock accuracy 100- NIC GPU 500nS PCI GPU • All server NICs are PTP enables and has at least two NIC port , One dedicated for DATA Network transport and the other is for PTP, trigger, PC- backend DAQ monitoring …. • GPU, easy to program, almost zero R&D 1.Most of the components are off-shell, minimum R&D and maximum flexibility 2.It is cost effective, low maintenance and rapid R&D with minimum usage of FPGA and related custom build components. 3.Commercial availability of Optical C&D/WDM Transceiver/Ethernet Optical Interfaces 100Gb/s ( to 400Gb in near future) 4.Very low power consumption ( in-detector cave ) 5.Extremely modular and scalable 7
GPU based architectures : : proof of f concept • What is the PTP accuracy : • We investigating PTP in Oxford, Created PTP a setup with two nodes and boundary switch • GPU concept: • Is it possible ? • How many GPUs we need ? • What is the bottle neck ? PTP test set-up in Oxford • How difficult is GPU programming? • ….. • Please listen to Phil’s Talk 8
Back up slides 9
Architecture; Tim ime and clo lock dis istribution • What is the clock and time distribution requirements? • Assuming feeding 1 clock line and one time stamp for each APA • What is clock phase difference between two APA (and SSPs)? • What is the time (stamp) accuracy per APA and SSP? • Sample rate is 2Ms/S (500ns), what if the 2 APA are out of phase for clock and time stamp in ring buffer? • “DUNE can tolerate a few tics ambiguity in clock and time stamp between APAs“ ? • We need a 50MHz! stable clock distribution generated by a atomic clock. 10
Architecture; PTP Tim ime dis istribution • Providing the time to stamp the ring buffer: 1. White Rabbit sub-nano second accuracy but very expensive. 2. Precision Time Protocol ( PTP ), 100ns to 500nS accuracy, almost all servers’ NIC and FPGA like ZYNQ are PTP enabled, only we need a PTP enabled switch to distribute the GPS + atomic global time. • This can be combined with Bristol's innovative clock distribution proposal to have simple clock and timing distribution. • Trigger can be send through TCP/IP data stream since we have a little relaxed letancy due to long ring buffer 11
Recommend
More recommend
