ProtoDUNE prototype TPC self-trigger Philip Rodrigues University of - PowerPoint PPT Presentation

ProtoDUNE prototype TPC self-trigger Philip Rodrigues University of Oxford 24 September 2019 1

A self-triggered muon parallel to the APA face Run 9619, event 6 (timestamp 0x1168a5f24e41158, 2019-09-09 15:41:35 UTC) Z view U view V view 9700 Offline channel number 9600 9500 9400 9300 4600 4500 Offline channel number 4400 4300 4200 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 Time (tick) Time (tick) Time (tick) ◮ Read out 1000 ticks before and after the trigger time 2

TPC-based self-trigger: why and how ◮ Why: Stepping stone to DUNE FD, which will need a self trigger ◮ How: ◮ Proof-of-concept; more concerned with data flow than physics ◮ Make the simplest thing that works; iterate if necessary ◮ Work in the existing ProtoDUNE DAQ, with incremental changes ◮ Downstream (lower data rates) shouldn’t impose requirements on upstream (higher data rates, performance more important) 3

System overview ptmp messages FELIX BR FELIX BR FELIX BR x10 Trigger cand’te BR APA 5 ◮ Implement code inside artdaq board readers: ptmp messages ptmp messages FELIX BR FELIX BR FELIX BR ◮ Advantages: integration with run control, log x10 Module-level file handling, saving of raw data and metadata trigger BR ◮ Trigger primitives and candidates are sent Trigger cand’te BR APA 6 artdaq fragment using ( from Brett Viren: https://github.com/brettviren/ptmp ) ptmp messages FELIX BR ◮ Data structures ◮ Message passing Routing ◮ Algorithms Master/DFO Trigger cand’te BR APA 4: on-host ◮ Not shown: SSPs don’t participate in generating triggers, but will provide data in ptmp messages Hit-finding FPGA Hit-sending BR response to a trigger request Trigger cand’te BR APA 4

ptmp data structures, message passing and algorithms ◮ TrigPrim : TPC hit with channel, start time, time span, ADC sum, peak ADC (unused so far), error flags (unused so far) ◮ TPSet : a container for TrigPrim s, with count, detector ID, creation time, time/channel span, actual list of TrigPrim s ◮ TPSet s can be passed as ZeroMQ messages over network or in-process. Fast, configurable, alternatives for handling backpressure (drop or wait) ◮ TPWindow algorithm repackages TrigPrim s into fixed-time windows ◮ TPZipper algorithm aggregates multiple TPSet message streams in time order, with (soft) maximum latency guarantee 5

Trigger primitive (hit) finding Run 9619, event 6 (timestamp 0x1168a5f24e41158, 2019-09-09 15:41:35 UTC) 400 300 ADC (arbitrary offset) 200 100 Ch. 9580 0 Ch. 9300 100 Ch. 9702 200 0 250 500 750 1000 1250 1500 1750 2000 Time (tick) ◮ Simple hit finding running in CPUs on FELIX BR hosts 1. Decode WIB format, select collection channels 2. Find pedestal and pedestal variance 3. Apply finite impulse response noise filter 4. Sum charge above threshold ◮ Uses about 60% of a CPU core per link (10/APA) 6

Trigger candidate (cluster) finding Figure: Jon Sensenig ◮ Trigger candidates are found per-APA ◮ Take 50 µ s windows, find groups of hits contiguous in channel (with up to 4-wire gap) ◮ Generate TC if channel range of largest group is 100 or more 7

Module-level trigger algorithm Figure: Jon Sensenig ◮ Stitch together TCs from individual APAs ◮ Make a sliding window of one drift time, look for consistent slope (∆(time) / ∆(channel)) between TCs ◮ Trigger if stitched TCs result in 350 hits or more in each APA 8

It works! Run 9619, event 6 (timestamp 0x1168a5f24e41158, 2019-09-09 15:41:35 UTC) Z view U view V view 9700 Offline channel number 9600 9500 9400 9300 4600 4500 Offline channel number 4400 4300 4200 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 Time (tick) Time (tick) Time (tick) ◮ Read out 1000 ticks before and after the trigger time 9

Another event Run 9406, event 4 (timestamp 0x11647d02ca488f4, 2019-08-23 17:07:39 UTC) Z view U view V view 9700 Offline channel number 9600 9500 9400 9300 4600 4500 Offline channel number 4400 4300 4200 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 Time (tick) Time (tick) Time (tick) ◮ Read out 1000 ticks before and after the trigger time ◮ Event from lower-purity period 10

Characterization: latency and CPU usage Run 9413 10 3 10 2 Frequency 10 1 10 0 0 20 40 60 80 Trigger Latency (ms) ◮ Trigger latency, defined as time between data with timestamp T arriving in FELIX BR, and trigger request for data with timestamp T arriving in FELIX BR. One histogram per link ◮ ∼ 3000 triggers in this run, no latencies close to buffer depth ( ∼ 1s) ◮ CPU usage: TODO, but approx 2–3 cores per APA for work downstream of hit-finding ◮ 11

Next steps ◮ Incorporate hit finding into on-host FELIX config ◮ Improve characterization/monitoring. In particular, check rate of dropping data (better be negligible!) ◮ Handle multiple trigger candidate algorithms: more like DUNE ◮ Michel electron trigger. Work from Columbia University ◮ Write technical paper (NIM or JINST maybe) ◮ Investigate induction-wire hit finding 12

Next steps: induction wire hit finding ◮ Could open the possibility of smaller ROI readout in DUNE ◮ Easy first test: run collection-wire algorithm on induction wires ◮ Works, for the appropriate value of “works”. Ran out of cores at 4 links (out of 10) ◮ Nice idea from ICARUS collaborators for induction wire preprocessing: keep a running sum of waveform, to convert bipolar to unipolar 13

Conclusions ◮ Demonstration of self-triggering on TPC data using an entirely software-based trigger chain ◮ A base for more R&D towards DUNE 14

Backup slides 15

A self-triggered muon parallel to the APA face, with hits Run 9619, event 6 (timestamp 0x1168a5f24e41158, 2019-09-09 15:41:35 UTC) Z view U view V view 9700 Offline channel number 9600 9500 9400 9300 4600 4500 Offline channel number 4400 4300 4200 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 0 250 500 750 1000 1250 1500 1750 2000 Time (tick) Time (tick) Time (tick) ◮ Read out 1000 ticks before and after the trigger time 16

Induction wire hit finding ◮ Each link uses two threads for hit finding: one to decode the funky WIB format and find collection-wire hits (as before), and another to find induction-wire hits. They each get a core to themselves, and use 75% and 65% CPU respectively ◮ Ran out of cores on the FELIX BR machine at 4 links. Expect improvements with a more favourable input data format. Also might just sum adjacent ticks to halve the data rate (there’s no information at frequencies that high anyway) 17

ptmp data structures ◮ Important distinction between “data time” and “computer time”: ◮ Data time is 50 MHz ticks from the timing system ◮ Computer time is microseconds according to a computer’s system clock ◮ TrigPrim , a trigger primitive: channel Offline channel number tstart Start time (data time) tspan Time span (data time) adcsum Sum of hit ADCs adcpeak Peak ADCs (not filled by CPU hit finding yet) flags Error flags. Not used yet ◮ TPSet , a set of connected trigger primitives: count Sequential count detid Which detector portion represented (in PDSP, this is FELIX link) created Time this TPSet created (computer time) tstart Earliest tstart of member TrigPrim s (data time) tspan Time span of member TrigPrim s (data time) chanbeg Lowest channel of TrigPrim s in set chanend Highest channel of TrigPrim s in set totaladc Total ADC of TrigPrim s in set tps The actual list of TrigPrim s ◮ Full details: https://github.com/brettviren/ptmp/blob/master/src/ptmp.proto ◮ count field means that lost data can be identified, but that’s not done yet ◮ By convention, empty TPSet s are not sent 18

ptmp message passing ◮ TPSet s are (de)serialized using protobuf from Google and the data sent using ZeroMQ ◮ Relevant features of ZeroMQ: ◮ Messages can go via network or efficiently within process. Just via a config change ◮ Can “subscribe” to messages in standalone code for debugging/data dumping ◮ Alternatives for handling backpressure: make upstream wait (PUSH/PULL) or drop messages (PUB/SUB) ◮ Good performance: millions of msgs/s, sub-ms latencies 1 1 http://zeromq.org/results:10gbe-tests-v432 19

ptmp algorithms: TPWindow TPSet 3 TPSet 2 TPSet 1 TPSet 2 TPSet 1 TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TPWindow TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] span=100 TrigPrim [t0, t1] TrigPrim [t0, t1] offset=0 TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] TrigPrim [t0, t1] ◮ Repackage TrigPrim s into new TPSet s with start (data) times in windows starting on a fixed boundary ◮ Input hits not required to be sorted by start time. Hits are buffered for a configurable time before being sent out. Late hits are dropped ◮ Full details: https://github.com/brettviren/ptmp/blob/master/docs/tpwindow.org 20