Gigabit Ethernet Gigabit Ethernet implementation for - PowerPoint PPT Presentation

Gigabit Ethernet Gigabit Ethernet implementation for implementation for FPGAs FPGAs Grzegorz Korcyl - Jagiellonian University, Kraków Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

Outline Motivation 1. 2. General structure GbE Buffer a) Packet Constructor b) Frame Constructor Frame Constructor c) c) Frame Transmitter d) Resources utilization e) 3. Results Setup 1 a) Setup 2 b) 4. Summary 5. Future plans Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

1. Motivation Addon + GbE GbE Switch 1 Evt 1 Evt 2 Evt 2 Evt 3 Evt 3 Evt builder builder builder � The way to transport large amounts of data at high speed � Protocol with routing capabilites needed � UDP over GbE is going to be implemented on HUB2, MDC, Shower addons Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2. General Structure � TrbNet – Hades trigger, slow control and data transport network system � GbE Buffer gathers subevents from TrbNet and prepares packet construction construction � Packet Constructor creates a HadesTransportUnitQueue � Frame Constructor fragments data and creates Ethernet frames � Frame Transmitter transmits each frame Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2. General Structure TrbNet Subevent 1 Subevent 2 Subevent 3 Subevent 4 GbE Buffer Message Message Packet Constructor Packet 1 Packet 2 Packet 3 Frame Constructor Frame 1 Frame 2 Frame 3 Frame 1 Frame 2 Frame 3 Frame 1 Frame 2 Frame Transmitter ETHERNET Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2.a GbE Buffer � Interface between TrbNet and GbE logic � Buffers incoming data � Responds back to readout request � Allows configuration via slow control � Allows configuration via slow control � Pepares the data for Packet Constructor � Decides about the start/close a packet � Selects event builder address basing on the readout parameteres � Adds additional internal subevent headers Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2.b Packet Constructor � Constructs HadesTransportUnitQueue: � Several subevents in one queue � Consistency check by termination bytes � Right now up to 64kB in total � Minimum overhead (one subevent case) is 56B per queue Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2.c Frame Constructor � Constructs IP frames: � Fragments the whole queue into 1.5kB (MTU) parts and encapsulates with network headers � 42B of headers overhead per frame Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2.d Frame Transmitter � Uses two IP Cores to transmit bytes over fiber: Tri-Speed MAC 1. Transmit state machine Transmit state machine � � Insertion of pause frames � Addition of Ethernet preamble and CRC � SGMII/GbE PCS 2. Interface between TS MAC and SERDES � Link state machine � Autonegotiation � Conversion to 8b/10b � Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2.e Resources utilization � Built on Lattice ECP2M LFE2M100E Resource Utilization Percentage Slices Slices 4256 out of 47376 4256 out of 47376 8% 8% LUTs 5984 out of 95000 6% Blocks RAM 76 out of 288 (~150kB) 26% � Mainly consumes RAM blocks inside FPGA Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

3.a Results – setup 1 � One data source – dummy front-end � HUBv2 � Netgear optical GbE switch � One event builder � Triggering by TRB Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

3.a Results – setup 1 � No pipelining, single packet construction at a time limits throughput to half of nominal link bandwidth � Event rate limited by single trigger in the system at a time Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

3.b Results – setup 2 � 5x ADCM data sources � HUBv2 � Longshine optical GbE switch � One event builder � Triggering by generator Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

3.b Results – setup 2 � Packet size 38.8kB � Accepted event rate 1.25kHz and 49MB/s throughput � Long-lasting and stable connection Old DAQ New DAQ Factor Bandwith to Event builder 6 MB/s 150 MB/s 25x faster Event rate / sector 50 Hz 1.25 kHz 25x faster Throughput / sector 1 MB/s 49 MB/s 50x faster Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

4. Summary Construction of HadesTransportUnitQueues of the 1. size up to 64kB Transmission of packets at satisfying speed 2. Multiple subevents combined into one queue Multiple subevents combined into one queue 3. 3. Configuration via slow control 4. Autonegotiation 5. Dynamic event builder selection 6. � Basic, ready to use GbE component is ready Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

5. Future Plans 1. Stabilize fast packet construction Pipeline packet construction in order to reach the full Gb � Large queues fragmentation into multiple UDP packets 2. � Ability to transport queues (subevents) larger than one UDP packet (64kB) packet (64kB) � Test different MTU sizes (Jumbo packets) in order to find the optimal configuration 3. Control of the transmission status and statistics � Generate status information and create systems to handle unexpected situations � Full featured GbE component planed to be ready in May Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010