DUNE DAQ Brainstorming E. Hazen October 31, 2017 Slide 1/9 - PowerPoint PPT Presentation

DUNE DAQ Brainstorming E. Hazen October 31, 2017 Slide 1/9 2017-10-31 E. Hazen et. al.

Brief Review of the Numbers Item Channels No/APA No/TPC bits/s ea Channel 1 2,560 384,000 24 Mb/s TDC Link 32 80 1200 768 Mb/s ROB 128 20 300 3.1 Gb/s APA 2560 1 150 61.4 Gb/s TPC 384,000 0.05 1 9.2 Tb/s No encoding overhead included The current cold elx plan suggests four 1 Gb/s links per ROB This is a total of 1200 links per TPC or 4800 for 4 TPCs. Slide 2/9 2017-10-31 E. Hazen et. al.

Readout Schemes... some choices 80 x 1Gb/s 80 x 1Gb/s Ethernet copper (simplex) (duplex fiber fibers) COTS FPGA APA Hardware Board I I Fiber Tx 2560 ch 61.4 Gb/s (GPU / CPU) (ATCA?) Proprietary mux'd fibers (simplex) 80 x 1Gb/s 10 x 10 or 4 x 25 Ethernet copper (duplex fibers) COTS FPGA FPGA APA II Hardware Board Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) (ATCA?) Ethernet 10 x 10Gb/s 80 x 1Gb/s Or 1 x 100Gb/s copper (duplex fibers) COTS FPGA APA Hardware III Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) Slide 3/9 2017-10-31 E. Hazen et. al.

Scheme I Simple WIB with 1Gb/s fiber Tx Simplex 1Gb/s (unidirectional) fibers to underground FPGA Rx 1Gb/s links with MGT or generic I/O ◮ Pros: ◮ Minimum power at cryostat (??) 80 x 1Gb/s 80 x 1Gb/s Ethernet copper (simplex) (duplex fiber ◮ Minimum noise at cryostat (??) fibers) COTS FPGA APA Hardware Board I I Fiber Tx 2560 ch 61.4 Gb/s (GPU / CPU) ◮ Minimum engineering required for board (ATCA?) Proprietary mux'd design fibers (simplex) 80 x 1Gb/s 10 x 10 or 4 x 25 Ethernet copper (duplex fibers) COTS ◮ Single layer of FPGA required in data FPGA FPGA APA II Hardware Board Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) (ATCA?) path Ethernet 10 x 10Gb/s 80 x 1Gb/s Or 1 x 100Gb/s ◮ Cons: copper (duplex fibers) COTS FPGA APA III Hardware Board 2560 ch ◮ Lots of fibers to run 61.4 Gb/s (GPU / CPU) (on-det.) ◮ FPGA board must be underground (or run 4800 fibers up shaft) Slide 4/9 2017-10-31 E. Hazen et. al.

Scheme II WIB with FPGA multiplexing (ala ProtoDUNE) Output 10-40 Gb/s unidirectional fibers (could be long-haul to surface or even Lead) Second processing FPGA (on surface) ◮ Pros: 80 x 1Gb/s 80 x 1Gb/s Ethernet copper (simplex) (duplex fiber fibers) FPGA COTS APA Board Hardware ◮ Can spy on data for debugging I I 2560 ch Fiber Tx 61.4 Gb/s (ATCA?) (GPU / CPU) ◮ Reduced fiber count 8x or 20x Proprietary mux'd fibers (simplex) 80 x 1Gb/s 10 x 10 or 4 x 25 Ethernet copper ◮ Most processing can be moved to surface (duplex fibers) FPGA FPGA COTS APA II Board Board Hardware 2560 ch ◮ Independent of processing technology 61.4 Gb/s (GPU / CPU) (on-det.) (ATCA?) Ethernet ◮ Cons: 10 x 10Gb/s 80 x 1Gb/s Or 1 x 100Gb/s copper (duplex fibers) COTS FPGA APA Hardware ◮ More noise and power at cryostat III Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) ◮ More engineering Slide 5/9 2017-10-31 E. Hazen et. al.

Scheme III WIB with FPGA multiplexing and Ethernet Output 10-40 Gb/s unidirectional fibers (could be long-haul to surface or even Lead) Std. network protocol (Ethernet) TCP/IP ( simplified ) or UDP ◮ Pros: ◮ Can spy on data for debugging 80 x 1Gb/s 80 x 1Gb/s Ethernet copper (simplex) fiber (duplex ◮ Reduced fiber count 8x or 20x fibers) COTS FPGA APA Hardware Board I I Fiber Tx 2560 ch 61.4 Gb/s (GPU / CPU) (ATCA?) ◮ Most processing can be moved to surface Proprietary mux'd ◮ All hardware past WIB is COTS fibers (simplex) 80 x 1Gb/s 10 x 10 or 4 x 25 Ethernet copper (duplex fibers) COTS FPGA FPGA ◮ One board to design APA II Hardware Board Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) (ATCA?) ◮ Cons: Ethernet 10 x 10Gb/s 80 x 1Gb/s Or 1 x 100Gb/s copper (duplex ◮ All hardware past WIB is COTS fibers) COTS FPGA APA III Hardware Board 2560 ch 61.4 Gb/s (GPU / CPU) (on-det.) limits FPGA processing options ◮ More noise and power at cryostat ◮ More engineering Slide 6/9 2017-10-31 E. Hazen et. al.

Scheme III Feasibility Case study: AMC13XG MicroTCA Module [1] ◮ Developed originally for CMS (100’s installed) ◮ Two firmware builds: ◮ Original CMS version (10 Gb/s proprietary protocol) ◮ Muon g-2 version (10 Gb Ethernet TCP/IP) ◮ Throughput of within a few % of line speed Input Links 5 Gb/s x 12 (uTCA) Simplified Link Tx 10 GbE FIFO TCP/IP DAQ Fiber (in AMC) Simplified 10 GbE TCP/IP DAQ Fiber Event Simplified Builder 10 GbE TCP/IP DAQ Fiber L1A DDR3 1600MT/s TTC FIFO SDRAM (6.4 GB/s) IPbus control / monitor / local DAQ GbE [1] “The AMC13XG: a new generation clock/timing/DAQ module for CMS MicroTCA”, http://dx.doi.org/10.1088/1748-0221/8/12/C12036 Slide 7/9 2017-10-31 E. Hazen et. al.

A possible “Scheme III” WIB Bandwidth required (1 40:10 mux): 40 Gb/s (in+out) = 80 Gb/s (2.5 GHz @ 32 bits) This might be a bit aggressive with the DRAM, but easy to increase BW DDR4-2666 x 32 5x 10 Gb/s 40x 1.0 Gb/s Ultrascale+ class FPGA 40x 1.0 Gb/s 5x 10 Gb/s (1 or 2) DDR4-2666 x 32 Slide 8/9 2017-10-31 E. Hazen et. al.

Summary ◮ I believe that sending the data to the surface must be cheaper in the long run (having worked on 6 underground experiments now!) ◮ It is (I assert) feasible using existing techniques to multiplex all data onto Ethernet in a simple way ◮ I believe this need not introduce noise into the cryostat if care is taken at the feedthru ◮ The main point of this presentation is to provoke discussion... Slide 9/9 2017-10-31 E. Hazen et. al.