ProtoDUNE/DUNE timing system and CCM Stoyan Trilov, University of - PowerPoint PPT Presentation

ProtoDUNE/DUNE timing system and CCM Stoyan Trilov, University of Bristol CCM meeting 13/11/2019 1

Outline ▪ ProtoDUNE I (overall) startup ▪ Timing system control layers – CCM – Control software – Firmware/hardware ▪ Behaviour of timing system: current (ProtoDUNE I) → future ( ProtoDUNE II/DUNE) – Timing master – Timing endpoint – GPS signal loss, and recovery – Timing master hot-swap – Commands for synchronisation signal 13/11/2019 2

ProtoDUNE I (overall) startup ▪ Getting ProtoDUNE I to “RUNNING” state requires interplay between all systems – Timing – DAQ – Trigger – Front-end electronics ▪ Should the timing system live outside the FSM in ProtoDUNE II/DUNE? – i.e. almost like infrastructure, e.g. power – Not currently the case in ProtoDUNE 13/11/2019 3

Timing system control layers for ProtoDUNE II/DUNE Debug and CCM development • Merely a proposal/idea • Many of possible solutions to the problem Timing system control • Not trivial to define interfaces software • Everyone has to agree Commands • and follow through Information Debug information/commands Hardware/firmware 13/11/2019 4

Timing system/CCM functionality ▪ Holds timing system configuration Debug and CCM – How many endpoints (with their UID) are expected to be found? development – Are all expected endpoints supposed to be functioning correctly? – Delay value for each endpoint ▪ Sends commands to timing system via intermediate control layer Timing – Configure these set of endpoints right now system control ▪ Receives distilled information from timing system software – Status of GPS signal lock – Number of found endpoints and their status ▪ Common interface for SP and DP timing systems? Hardware/ firmware 13/11/2019 5

Timing system/control software functionality ▪ Has knowledge of low level hardware/firmware API Debug and CCM – Can query and report low level information from timing system development – Send low level commands to timing master, and endpoints (via master?) – Useful for lower level debugging and during development/commissioning ▪ Able to aggregate status information of timing system Timing – Condense multiple datapoints into single statuses, e.g. OK, WARNING, ERROR system – Automate certain functionality, e.g. recovery of lost endpoints, periodic delay control adjustments for end points, timing master hot-swapping software ▪ Receive/execute commands from CCM and debug interface (humans) – Reset and configure particular endpoint(s) Hardware/ firmware 13/11/2019 6

Timing system/fw+hw functionality ▪ Receives/executes commands from the control software Debug and CCM – e.g. commands implemented in firmware development ▪ Low level state machine – Different and invisible to CCM, e.g. waiting for CDR lock ➢ However information still available via debugging interface Timing ▪ Performs delay alignment system control – Measures correct delay to be used, and applies it software Hardware/ firmware 13/11/2019 7

Timing master: ProtoDUNE I ▪ ProtoDUNE I timing master implemented as an AIDA TLU – It does not use a GPS oscillator – Derives 10 MHz signal from DP WR system (WR-LEN) ▪ Before entering “RUNNING” state, the master needs to – Load correct configuration, IP address, etc. – Receive relevant commands. e.g. “start” ▪ Possible causes of malfunction – Loss of WR 10 MHz signal → switch to internal oscillator ➢ System continues to function, but clock will drift from GPS time 13/11/2019 8

Timing master: ProtoDUNE II/DUNE ▪ Envisage timing system in ProtoDUNE II/DUNE in an “always on” state – i.e. if there is power, there should be an active timing system – Master/spare selection done by software (DUNE only) ▪ On startup either from cold start or reset – Wait for stable 10 MHz from GPS/and subsequent lock? ➢ GPS oscillator startup and monitoring? – Power on/Initialise fanout AMC/FMCs? – Signal lock for each unit? – Configure endpoints 9

Timing endpoint: ProtoDUNE I Startup sequence State Description ▪ Firmware FSM for an endpoint shown on the left 0x0 Standing by – Full version would be hidden to CCM? 0x1 Waiting SFP for signal – Only error and condensed status states propagated 0x2 Waiting CDR lock upwards? 0x3 Waiting for good frequency check ▪ In case of an endpoint malfunction 0x4 Waiting for comma alignment – Board state machine moves to error 0x5 Waiting for 8b10 decoder good packet ▪ In theory to recover an endpoint, only actions against the endpoint are required 0x6 Waiting for time stamp initialisation – e.g. reset, or firmware reload – Not actually implemented in ProtoDUNE due to the lack of Waiting for delay setup two-way communication with the endpoints 0x8 Ready ➢ Photon system laser issue Error states 0xc Error in Rx 13/11/2019 0xd Error in time stamp check 0xe Error in physical layer after lock 10

Timing endpoint: ProtoDUNE II/DUNE ▪ In case of an endpoint failure, an error state is raised – Flag propagated up to CCM by endpoint – CCM keeps track for how long the endpoint is down ▪ Automated recovery procedure – Attempt recovery ➢ Reset, firmware reload, etc. – What happens if automated recovery procedure failure?→ manual intervention, … ▪ Use mock endpoints to diagnose endpoint issue – Commissioning 13/11/2019 11

Loss of GPS signal: ProtoDUNE II/DUNE ▪ ProtoDUNE II – It is envisaged to have its own GPS disciplined oscillator providing 10 MHz and time signals to the timing system – GPS signal loss may not necessarily mean clock drift, depends on duration ▪ DUNE – It will have 2 redundant GPS disciplined oscillators providing 10 MHz and time signals to 2 redundant timing systems (hot swap) – What happens in case of GPS signal loss in active system? May not have to trigger a swap immediately. – Use Rb oscillator + software to cross-check lock of the 2 GPS oscillators ▪ GPS oscillator is certainly going to output a host of monitoring information ( SNMP?) – GPS signal lock – Clock frequency information – FSM? 13/11/2019 12

DUNE redundant timing system 13/11/2019 13

DUNE timing system hot-swap ▪ The two redundant timing systems transmit into a passive combiner – Only one system transmits at any one time, one is active, one is in standby – Allows firmware upgrades/tests without downtime ▪ How do you decide if the redundant system should take over? – Software control system which monitors both systems and detects faults? – Cross-communication between the two systems? – What level of fault triggers the swap? Warnings vs errors? – Redundant systems is not a new concept, principles and protocols already developed → research ▪ Testing hot-swap functionality will be a major and important task 13/11/2019 14

Sync commands ▪ ProtoDUNE I Sync commands – Regular timestamp messages sent once ~1s TimeSync 0x0 Echo 0x1 – Sync signals are currently enabled following reset SpillStart 0x2 ➢ done via (pdtbutler), i.e. pdtbutler io PDTS_TERTIARY reset SpillStop 0x3 RunStart 0x4 RunStop 0x5 ▪ ProtoDUNE II/DUNE WibCalib 0x6 – When are sync commands enabled? SSPCalib 0x7 – Add/remove commands from current list? FakeTrig0 0x8 FakeTrig1 0x9 FakeTrig2 0xa FakeTrig3 0xb BeamTrig 0xc NoBeamTrig 0xd ExtFakeTrig 0xe 13/11/2019 15

ProtoDUNE I → ProtoDUNE II/DUNE summary ▪ Endpoint calibration and monitoring testing – Requires fix of PDS laser issue – Can also be tested in an lab environemnt ▪ Development/test of software allowing endpoints to dropout and rejoin DAQ ▪ GPS oscillator setup and testing – Prototyping DUNE-like system. mTCA, etc. 13/11/2019 16