Single-phase TPS Warm Interface Some thoughts E. Hazen, M. Johnson, R. Van Berg 2016-02-08 E. Hazen - DUNE Warm Interface 1/18
Focus on... 1. Slow Controls 2. Clock and synchronous controls 3. DAQ 2016-02-08 E. Hazen - DUNE Warm Interface 2/18
Slow Controls ● Requirements (actually very few!) – Must provide read/write access to registers – Must be reliable ● Must tolerate cold+warm ops, 25m cable, DC offsets etc – Should be simple to implement cold+warm ends – Should be easy to integrate to custom/COTS system ● Proposed Implementation (COLDATA interface): – LVDS electrical standard – Three LVDS pairs: CLK in, Data in, Data out – “I2C-Like” protocol, – Details t.b.d. (FNAL will provide a proposal) ● By far the easiest way to accommodate this is with an FPGA or microcontroller on the warm interface board – Downstream interface would be a COTS standard specified by the slow controls group (RS-232, Modbus, CAN bus, Ethernet, whatever) 2016-02-08 E. Hazen - DUNE Warm Interface 3/18
Clock and Controls ● Requirements (to support measurements of): – TPC times to ¼ of ADC conversion time (125ns) (to be better than sqrt(12) ADC sampling time resolution) – Provide “fast” control path to TPC front-end (ADC convert and a few other anticipated signals) – Link to Universal Time to tag beam arrival (sub- µ s) ● COLDATA interface (proposed) – One pair with 50 MHz clock – One pair with encoded controls – Details t.b.d. (FNAL will provide a proposal) ● Upstream (distribution system, only a proposal) – Central clock module near top of cryostat – Above ground GPS receiver with ~ 3 fibers down to cryostat – Distribution system for clock, controls cables and to photon detectors 2016-02-08 E. Hazen - DUNE Warm Interface 4/18
Timing Considerations ● Ideally we know the time of each ADC sample in the system to < 1 conversion period ● COLDATA requirements: – Timestamp each data packet with 16-bit count, incremented every (50 MHz) clock cycle – Reset/check 16-bit count on receipt of RESET command, if count ≠ 0xNNNN on RESET, flag error – RESETs are counted elsewhere in clock system and on warm interface FPGA to provide many more bits of timestamp. 2016-02-08 E. Hazen - DUNE Warm Interface 5/18
DAQ – Cold HW Review 2 MSPS Two output streams 200Mb/s per ADC ADC 16 ch 12 bits Raw sampled data volume: 16 ch * 12 bits * 4 ADC * 2 MSPS ADC = 1536 Mbits/s 16 ch 12 bits Two 1.2 Gb/s COLDDATA ADC adds 4 bits per stream per output streams ASIC sample; this is an additional: (8b10b code) ADC 8 streams * 4 bits * 2 MSPS 16 ch 12 bits = 64 Mbits/s 1536 + 64 = 1600 Mbits/s ADC 16 ch 12 bits This is ½ of a Front-End Board (64 channels) 2016-02-08 E. Hazen - DUNE Warm Interface 6/18
DAQ Warm Interface (from Cold perspective) ● Receive digitized data: – ADC output (each 500ns): ● 16 Channels, 12 bits on two output streams – 96 bits plus 4 header bits = 100 bits per stream ● COLDATA inputs 8 such streams, so 800 bits every 500ns (1600 Mb/s) – COLDATA outputs (there are two): ● 1.2 Gb/s w/ 8b10b so 1.0 Gb/s payload (500 bits per 500ns) (Two links gives 2000 Mb/s) ● This provides 20% spare bandwidth (2000 / 1600) ● Transmit to DAQ: – Few “requirements” here, but some thoughts: ● Avoid constraining the DAQ choice ● Allow for DAQ on the surface if that seems best 2016-02-08 E. Hazen - DUNE Warm Interface 7/18
DAQ Interface Considerations ● 25m cables seriously degrade the 1.2Gb/s – At or below threshold for LVDS / CML inputs ● Ideally one would use a low-cost FPGA to receive and aggregate these links to e.g. 10Gb or 40Gb fibers – Receiving with normal I/O on FPGA is cheapest (but may not work, needs a bit of R&D) – Receiving with high-speed SERDES (as on SBND board) would work, but more expensive – In either case an active equalizer circuit is likely required, at least on the long cables 2016-02-08 E. Hazen - DUNE Warm Interface 8/18
One Proposal 1.2 Gb 10 GbE Up shaft Cu Fiber 10 GbE to surface to DAQ COLDDATA FPGA- DWDM Based Demux x8 10 GbE Data X 32 Merge DWDM COLDDATA Multplex 32x Total 1500 links Total 50 links Total 12k links 2016-02-08 E. Hazen - DUNE Warm Interface 9/18
FPGA Multiplex (conceptual design) Optional Generic IO SERDES active equalizer or high-speed SERDES (long cables only) 8-10x 1.2Gb LVDS Inputs 10Gb capable SERDES FPGA 10Gb optical transceiver (DWDM channel chosen) Kintex-7 Class or Altera equivalent (Entire block could be repeated using larger FPGA to minimise overall cost) Firmware combines 8-10 streams into a single 10Gb stream (RAM required (could be std. TCP/IP – firmware SDRAM for TCP/IP already exists for Muon g-2) buffering) 2016-02-08 E. Hazen - DUNE Warm Interface 10/18
FPGA Multiplex ● Quite similar to SBND board and other proposals ● Benefits: – Decouples DAQ from COLDATA protocol – Standard TCP/IP an easy option – Reduces link count by 8-10x – Can use DWDM to reduce fiber count such that entire DAQ can be on surface – Can include clock and slow controls in same box/board ● Possible Optimizations: – Use 40 or 100Gb output – Two-stage using inexpensive FPGA (e.g. Arria-V) to receive 1.2Gb, second stage to handle fast outputs – Implement cable equalizer on plug-in board 2016-02-08 E. Hazen - DUNE Warm Interface 11/18
DWDM (Dense Wavelength Division Multiplexing) ● Up to 40 x 10Gb streams on one single- mode fiber ● SFP fiber transceivers available on 40 different optical channels ● DWDM mux/demux is a passive, unpowered unit 2016-02-08 E. Hazen - DUNE Warm Interface 12/18
DWDM ● Benefits: – Could transport all data for one TPC to surface on ~50 duplex fibers (simplex if not TCP/IP) – In principle putting the DAQ on the surface would be less expensive ● Costs: – Must purchase (more) expensive fiber transceivers – Must purchase DWDM mux/demux units 2016-02-08 E. Hazen - DUNE Warm Interface 13/18
COLDATA Warm I/O Summary Three pairs CLK, Tx + Rx “I2C-Like” protocol Two Pairs CLK + encoded control Two pairs 1.2 Gb/s 8b10b out One packet per ADC conversion 2016-02-08 E. Hazen - DUNE Warm Interface 14/18
Summary ● It seems reasonable to implement a warm interface with one or more FPGAs near the flange with the following functions: – Slow controls interface: “I2C-like” to std protocol – Local distribution of clock and encoded controls with local decoding for time-stamping of data – Receive 1.2Gb LVDS links – Aggregate N x 1.2Gb links and retransmit on fiber at industry standard speed (10/40/100 Gb/s) 2016-02-08 E. Hazen - DUNE Warm Interface 15/18
Backup Slides 2016-02-08 E. Hazen - DUNE Warm Interface 16/18
Additional Concerns (outside scope of this meeting) ● We are concerned that reliability be addressed – For example, a COLDATA failure takes out a significant number of U, V and Y channels – Should we consider a different architecture with more links? COLDATA per ADC? ● We should ensure that the COLDATA control logic work correctly under fault conditions: – Shorted or spuriously driven inputs ● Should we add a provision for an analog monitor output for selected channels? 2016-02-08 E. Hazen - DUNE Warm Interface 17/18
Clock / Fast Controls ● Requirements on clock/controls system – Must distribute 50MHz master clock to all FEBs – Must distribute synchronous commands (precision = 1 clock tick) ● CONVERT every 500 ns ● RESET periodically ● TEST pulse ● Others.. – Must provide a monitoring scheme for distributed clock/controls ● Proposed Implementation (general features) – Two LVDS pairs carries clock, controls – Overall DC balanced transmission – Easy / unambiguous algorithm to decode – Minimum effect on clock jitter ● Details t.b.d. but FNAL will provide a detailed proposal 2016-02-08 E. Hazen - DUNE Warm Interface 18/18
Recommend
More recommend
