A data acquisition system for the Cerenkov Telescope Array Julien - PowerPoint PPT Presentation

A data acquisition system for the Cerenkov Telescope Array Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr

Camera server

Global architecture Telescope N Telescope 2 Telescope 1 Camera front end boards Camera front end boards Camera front end boards nN... n2... n1... 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 ... Camera Camera Camera Server Server Server N 2 1 Central server CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 3

Camera data flow PM event 1 event 2 event N Header Header Header PM 1 sampling data PM 1 sampling data PM 1 sampling data Front end board PM 2 sampling data PM 2 sampling data PM 2 sampling data PM 3 sampling data PM 3 sampling data PM 3 sampling data ... PM 4 sampling data PM 4 sampling data PM 4 sampling data PM 5 sampling data PM 5 sampling data PM 5 sampling data PM 6 sampling data PM 6 sampling data PM 6 sampling data PM 7 sampling data PM 7 sampling data PM 7 sampling data Whole Camera ~ 2000 PM -> 300 front end boards CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 4

Camera server ● Build event ● L2 trigger on camera server (L1 on front end) : ● CPU (SSE, AVX...) ● GPU ● Compress ? ● Send data to central server (array level) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 5

Data flow hypothesis ● ~ 2000 pixels camera ● L1 trigger rate : 10 KHz ● Size of sampling data for 1 PM : 144 bytes (16 bit * 72 samples) ● No data loss (all the L1 events are sent) Max theorical bandwidth = 10000 * 2000 * 144 = 2.88 GB/s = 23 Gb/s ● 7 detectors for each front end board : 300 boards/camera Each board generates a flow of 2880/300 = 9.6 MB/s = 77 Mb/s https://portal.cta-observatory.org/WG/ACTL/SitePages/Data%20Rates.aspx CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 6

Global architecture Ethernet link Camera Front end Server board CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 7

Camera infrastructure 300 * 1Gb/s To central To central server Ethernet links server OR 3 * 10Gb/s Front end boards Ethernet links SFP+ 7 stacked Powerconnect 6248 7*48 ports @ 1Gb/s One or several . Dell T7500 workstations . . CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 8

Dell Precision T7500 ● Two Intel Xeon X5650 (2.66GHz,6.4GT/s,12MB,6Cores) ● Memory : 24GB (6x4GB) 1333MHz ● Intel X520 DA2 10GbE Dual Port SFP+ Server Adapter, PCIe x8 ● Triple channel (maximum speed reached) ● QPI at 6.4 GT/s (maximum speed on the market) ● Memory DDR3-1333 ● 2 full speed full duplex 10 Gb/s links (PCIe x8 Gen 2) ● 1 PCIe x16 slot free (->GPU) and 1 PCIe x8 free (-> one more 10 Gbps adapter) ● SFP+ -> Copper or Optical link ~ 3500 euros CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 9

Dell Powerconnect 6248 ● 48 * 1 Gb/s ports ● Backplane 184 Gb/s ● 2 * 10 Gb/s SFP+ ports included 2 more 10 Gb/s optional ports ● Up to 12 switches stackable -> 576 ports ~ 1500 euros (with 2 * 10 Gb/s) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 10

Event builder

Why a prototype ? We need a prototype : ● To evaluate the maximum speed reachable ● To test several technologies ● To validate different approaches of the data processing ● To adapt our needs to what we can do CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 12

Our first approach ● High modularity to make adaptation to different front end electronics easier ● Multitask approach to divide the flow processing if needed ● Use of a standard Linux distribution but take control on scheduling and memory allocation ● Constrained electronics to reach the best performances (in a first time) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 13

Event builder Trigger FE1 FE2 FE3 FE4 FEN PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H H H H H Ev1 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 Ev2 H H H H H ... ... ... ... ... PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 Ev N H H H H H Copy Copy Copy Copy Copy Ev1 H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 Ev2 H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... Ev N H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 14

Data format : regular frame Level 2 triggering on camera server 1 frame : 1024 bytes s 4 e 2 Event n t 0 y 0 32 64 1 b header Level 1 Header flag FE Id Event number bytes 16 Time 1 Time 2 bytes 144 PM 1 sampling data bytes 144 PM 2 sampling data bytes 144 PM 3 sampling data bytes 144 PM 4 sampling data s 4 e PM 5 sampling data 4 t y 1 b s 4 e PM 6 sampling data 4 t y 1 b bytes 144 PM 7 sampling data CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 15

Data format : jumbo frame Level 2 triggering on camera server 1 frame : 8192 bytes s 4 e 2 Event n t 0 y 0 32 64 1 b header Level 1 Header flag FE Id Event number bytes 16 s 4 Time 1 Time 2 e 2 Event n+1 t 0 y 1 b bytes 144 PM 1 sampling data s 4 e 2 Event n+2 t 0 y 1 b bytes 144 PM 2 sampling data s 4 e 2 Event n+3 t 0 y 1 b bytes 8192 bytes 144 PM 3 sampling data s 4 e Event n+4 2 t 0 y 1 b bytes 144 s PM 4 sampling data 4 e Event n+5 2 t 0 y 1 b s s 4 4 e PM 5 sampling data e 2 Event n+6 4 t t y 0 1 y b 1 b s s 4 4 e e 2 Event n+7 PM 6 sampling data 4 t t 0 y y 1 1 b b bytes 144 PM 7 sampling data CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 16

Software overview : 1 st architecture N2 camera elements N1 camera elements Nx camera elements 1 1 1 Socket Socket Socket (blocking) (blocking) (blocking) packetReceived packetReceived packetReceived SYNC SYNC SYNC SYNC SYNC SYNC 1 1 1 taskReceiveDaq taskReceiveDaq taskReceiveDaq Packets Packets Packets ... PacketReceived PacketReceived PacketReceived Semaphore Semaphore Semaphore SYNC SYNC SYNC ... ... ... 1..n 1..n 1..n ReceiveDaqPacketBuffer ReceiveDaqPacketBuffer ReceiveDaqPacketBuffer taskBuildEventDaq taskBuildEventDaq taskBuildEventDaq C ComputeEvents N Y Semaphore S Events ... computeEventsSemaphore SYNC eventBuffer SYNC computeEventsSemaphore 1..n CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 17 taskComputeEventDaq

Software overview : 2 nd architecture N2 camera elements N1 camera elements Nx camera elements . . . 1 1 1 Socket Socket Socket (blocking) (blocking) (blocking) packetReceived packetReceived packetReceived SYNC SYNC SYNC S S S Y Y Y N N N C C C 1 1 1 Task Task Task ReceiveDaq ReceiveDaq ReceiveDaq + + + BuildEvent BuildEvent BuildEvent C ComputeEvents N Y Semaphore S Events ... computeEventsSemaphore SYNC eventBuffer SYNC computeEventsSemaphore 1..n taskComputeEventDaq CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 18

Stimulation configuration 10 * 2 * 1Gb/s 10 HP servers Ethernet links 2 * 10Gb/s Ethernet links SFP+ One Dell T7500 workstation CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 19

Stimulation room CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 20

First results : event builder 150 nodes (15 per HP server) sending data to interface 1 300 nodes simulated 150 nodes (15 per HP server) sending data to interface 2 Tests of the event building with varying packet size: 1 st Achitecture 2 nd Achitecture Jumbo frames (8192 bytes) : Jumbo frames (8192 bytes) : 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) with no loss with no loss CPU usage : 300 % (3 cores/12) CPU usage : 160 % (1.6 cores/12) Regular frames (1024 bytes) : Regular frames (1024 bytes) : 6,5 Gb/s (0,82 GB/s) with no loss 8 Gb/s (1 GB/s) with no loss CPU usage : 300 % (3 cores/12) CPU usage : 170 % (1.7 cores/12) Results obtained with standard libraries/drivers (Linux). Test of a direct I/O solution to improve small frames reception in progress CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 21

Integration in ACS The basic functions of the Event Builder are available from the ACS interface CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 22

Stimulator

Need for a real stimulator Need a stimulator to make : ● timing measurements on software ● real time validation ● algorithms validation ● trigger validation ● latency measurements on network ● front end boards and stimulator mix ● validate the complete acquisition chain CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 24

A data acquisition system for the Cerenkov Telescope Array Julien - PowerPoint PPT Presentation

A data acquisition system for the Cerenkov Telescope Array Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr Camera server Global architecture Telescope N Telescope 2 Telescope 1 Camera

Gamma-ray radiation from type IIb supernova remnants prospect for the Cerenkov Telescope Array

Composition study of ultra-high-energy cosmic rays based on the Telescope Array surface detector

TA Spectrum Summary : Energy Spectrum Measurements With The Telescope Array Detectors

Optimization of the Data Acquisition Software (PxSuite DAQ) for the Silicon Strip Telescope at

Telescope Array search for EeV photons Yana Zhezher (on behalf of Mikhail Kuznetsov), Oleg

Telescope Array: Status and Upgrades John Belz University of Utah Snowmass on the Mississippi

TA Anisotropy Summary Eiji Kido for the Telescope Array Collaboration 1 Outline

CHERENKOV TELESCOPE ARRAY FOR INTENSITY INTERFEROMETRY Dainis Dravins Lund Observatory, Sweden

The Cherenkov Telescope Array SST-1M camera prototype commissioning T .R.S. Njoh Ekoume*, C.

Ch Cherenkov Telescope Array: k T l A The Next Generation e e t Ge e at o Gamma-ray

Results from the Telescope Array Experiment Charlie Jui University of Utah TeVPA 2013 Irvine,

LOFAR DATA MANAGEMENT R. F. Pizzo ASTRON, December2 nd 2015 THE LOW FREQUENCY ARRAY KEY

ARA detector calibration with Telescope Array Electron Light Source Romain Gaior for the Chiba

Atmospheric calibration of the Cherenkov Telescope Array Jan Ebr for the CTA Consortium

AGN Physics with the Cherenkov Telescope Array A. Zech (for the CTA Consortium) LUTH,

The Telescope Array Low-Energy Extension (TALE) C. Jui For the TA/TALE Collaboration ISVHECRI,

Data Acquisition Chapter 2 Data Acquisition 1 st step: get data Usually data gathered by

Chapter 7 1 Data Acquisition 2 Fix SI units throughout, and remove a lot of the dword

Background Signals in FOS Data: Cerenkov and Scattered Light Michael R. Rosa 1,2 Abstract The

Skills, innovation, and interactive capabilities: the case of the square kilometre array telescope

Agreement between the Xmax distributions measured by the Pierre Auger and Telescope Array

The TAx4 experiment E. Kido for the Telescope Array Collaboration Institute for Cosmic Ray

EXTRAGALACTIC SOURCE POPULATION STUDIES AT VERY HIGH ENERGIES IN THE CHERENKOV TELESCOPE ARRAY

Network Telescope Data Analysis: IBR Monitoring telescope/darknets/darkspace, 22 Mar 11 Nevil