Parsa, Andrew Siemion, Dan Werthimer, Mel Wright Outline What is a - PowerPoint PPT Presentation

Jason Manley, Aaron Parsons, Don Backer, Henry Chen, Terry Filiba, David MacMahon, Peter McMahon, Arash Parsa, Andrew Siemion, Dan Werthimer, Mel Wright

Outline • What is a correlator? • Scalable packetized correlators: – The architecture – The hardware – The software – The cost • Closing thoughts • Walk through actual design • Questions and comments

Interferometry …

Basic idea V k ∑  V ik Z -n ∑ 90 °  V ii Amplitude Amplitude ∑  V jk Time Time ∑  V ik V i V i ∑ ∑  V ij  V ij Z -n ∑  V jk 90 ° 90 ° ∑  V jj ∑  V kk V j V j ∑ ∑ Z -n  V ij  V ij

“Actual” FX Correlator ∑ Z -n FFT ∑ Z -n FFT ∑ Z -n FFT

CASPER DSP backend concept Reconfigurable Polyphase Compute Cluster Filter Banks FPGA DSP PFB ADC Module FPGA DSP Module PFB . ADC Correlator . . FPGA DSP Module FPGA DSP Commercial off-the-shelf Module . Beamformers/ Multicast 10 Gbps (10GE Spectrometers or InfiniBand) Switch . FPGA DSP Module . FPGA DSP Pulsar timer Module . . . . . . PFB General-purpose CPUs ADC

Design Philosophy • Standardized processing hardware • Commercial interconnect • Asynchronous compute engines • Synchronization using common 1PPS • UDP output delivery over ethernet network • Correlator scales with your array

CASPER FX Architecture F Engine 0 X Engine 0 F Engine 1 X Engine 1 . . . . . . . . . 10GbE Switch F Engine N-1 X Engine N-1

Implementation F Engine 0 X Engine 0 F Engine 1 X Engine 1 . . . . . . . . . 10GbE Switch F Engine N-1 X Engine N-1

Architecture to hardware mapping Example 8 Antenna system iBOB BEE2 iBOB BEE2 user FPGA BEE2 user FPGA F Eng F Eng X Eng X Eng X Eng X Eng F Eng F Eng iBOB iBOB BEE2 user FPGA BEE2 user FPGA F Eng F Eng X Eng X Eng X Eng X Eng F Eng F Eng 10GbE Switch

F Engine Operations DDC Quantize Channelize Reformat X Engine ADC • Two F engines per iBOB • Dual polarization design • Currently uses ASTRO library • Currently processes data at native clock rate (<200MHz IBOB or < 400MHz ROACH)

Setup and Control Clocks: • X engines each run off independent clock – Sampling synchronized at F engines, but clock not distributed to X engines – Synchronized using global 1pps signal at ADCs • Propagated to X engines using out-of-band signaling on XAUI links – Headers labeling 10GbE Ethernet packet data – System control: separate 100Mbps Ethernet network on BEE2 • F engines configured from BEEs through XAUI links • Control packets: CASPER UDP framework on BEE2 control FPGA • Execute Python scripts for configuration, control and debugging •

F engine development • 2008: – Coarse delays (cable length compensation) – Fringe-stopping & fine delays – Walsh code generation and phase switching – Real sampling (low bandwidth) – Parallel streams (high bandwidth) • Future: – Ability to output subset of band – Spectral zoom modes

X Engine Operations 10GbE Buffer X Eng Accum F Engine • Using CASPER library • Scales with 2^N antennas • Fit as many X engines on an FPGA as possible (2x 16 ant on BEE2 usr)

Backend Software • UDP packets received • Currently received, parsed and saved in MIRIAD file format by single computer. • Computing requirements dependant on experiment; • Usually single computer ok: 128 antennas, 1 sec integrations, 2k chan = 512MB/s

Pending systems • Bench sys: 8ant, DP, 200MHz, 2k ch • PAPER: 128ant, DP, 100MHz, 2k ch • KAT-7: 8ant, DP, 256MHz, 2k ch • meerKAT: 80ant, DP, 1GHz, 16k ch • Bologna: 32ant, SP, 32MHz, 1k ch • GMRT: 32ant, DP, 400MHz, 4k-8k ch

How does it scale 1000000 100000 10000 1000 100 F Engines X Engines 10 1

FPGA Roadmap Xilinx Virtex Family 400 Logic Cells Thousands 300 330 200 200 100 100 43 0 2000 2002 2004 2006 • Processing power doubling every two years • V4 = ½ power requirements of V2Pro* * Manufacturers claim - Xilinx Inc.

Coming soon… • 10Gbps output optionally gives integrations ~10ms • More efficient use of hardware DSP slices • High speed, scalable, distributed data capture software • Walsh codes and phase switching • Phase rotation • 64 antenna design • Upgrade to 4096 channels • ROACH hardware: – <400MHz bandwidth – 16 384 channels – 128 antennas – no architectural changes

Questions and Comments Visit the CASPER correlator page: http://casper.berkeley.edu/wiki/index.php?title=Correlator Add your own requirements: http://casper.berkeley.edu/wiki/index.php?title=International_ Correlator_Collaboration Email me: jason_manley@hotmail.com

PFB-FFT response

Current uses Pocket Spectrometer • Using ATMEL ADC’s at 2 Gsamples/sec • Performing 4 real FFT’s in 1 (complex) biplex pipelined FFT module. • 2048 channels • Uses just 1 ADC, 1 IBOB, and your laptop.

ROACH block diagram