Processing Real-Time LOFAR Processing Real-Time LOFAR Telescope - PowerPoint PPT Presentation

Processing Real-Time LOFAR Processing Real-Time LOFAR Telescope Data on a Blue Gene/P Telescope Data on a Blue Gene/P John W. Romein John W. Romein Stichting ASTRON (Netherlands Institute for Radio Astronomy) Dwingeloo, the Netherlands ScicomP/SP-XXL'09 May 20, 2009 1 ScicomP/SP-XXL'09 May 20, 2009

LOw w F Frequency requency AR ARray ray LO  radio telescope  10–240 MHz  unexplored  dishes infeasible  ionospheric disturbance  new design ScicomP/SP-XXL'09 May 20, 2009 2 ScicomP/SP-XXL'09 May 20, 2009

A New Design A New Design  distributed sensor network  no dishes  O (10,000) antennas  omni-directional  concurrent observations  software telescope  flexible  requires supercomputer ScicomP/SP-XXL'09 May 20, 2009 3 ScicomP/SP-XXL'09 May 20, 2009

LOFAR Structure LOFAR Structure  hierarchical  receiver  (tile)  station  telescope  central core  Exloo  central processing  Groningen  real time  off-line ScicomP/SP-XXL'09 May 20, 2009 4 ScicomP/SP-XXL'09 May 20, 2009

LOFAR Science LOFAR Science  Epoch of Re-ionization  cosmic rays  extragalactic surveys  transients  pulsars ScicomP/SP-XXL'09 May 20, 2009 5 ScicomP/SP-XXL'09 May 20, 2009

Outline Outline  from wave to image  basics  receivers  stations  real-time Blue Gene/P processing  performance  off-line processing  image ScicomP/SP-XXL'09 May 20, 2009 6 ScicomP/SP-XXL'09 May 20, 2009

Reflectors vs. Phased Arrays Reflectors vs. Phased Arrays Physical delay Receiver Receiving array Combiner Artificial delay Output ScicomP/SP-XXL'09 May 20, 2009 7 ScicomP/SP-XXL'09 May 20, 2009

Beam Forming Beam Forming Physical  delay determines delay observation direction  beam forming = delayed addition Receiving array  diameter determines FoV  use earth rotation Combiner Artificial delay Output ScicomP/SP-XXL'09 May 20, 2009 8 ScicomP/SP-XXL'09 May 20, 2009

LOFAR Antennas LOFAR Antennas  two antenna types  Low-Band Antenna (10–80 MHz)  High-Band Antenna (110–240 MHz)  FM radio range not covered ScicomP/SP-XXL'09 May 20, 2009 9 ScicomP/SP-XXL'09 May 20, 2009

Low-Band Antennas Low-Band Antennas  10–80 MHz  dual polarized ScicomP/SP-XXL'09 May 20, 2009 10 ScicomP/SP-XXL'09 May 20, 2009

LBA Field LBA Field ScicomP/SP-XXL'09 May 20, 2009 11 ScicomP/SP-XXL'09 May 20, 2009

HBA Tiles HBA Tiles  110–240 MHz  dual polarized  4x4 receivers = 1 tile  analogue beam forming ScicomP/SP-XXL'09 May 20, 2009 12 ScicomP/SP-XXL'09 May 20, 2009

A Station A Station  48–96 LBAs  48–96 HBA tiles ScicomP/SP-XXL'09 May 20, 2009 13 ScicomP/SP-XXL'09 May 20, 2009

Station Cabinet Station Cabinet  station processing ScicomP/SP-XXL'09 May 20, 2009 14 ScicomP/SP-XXL'09 May 20, 2009

Remote Control Unit Remote Control Unit  2 LBAs + 1 HBA tile  filter  200 (or 160) MHz A → D conversion ScicomP/SP-XXL'09 May 20, 2009 15 ScicomP/SP-XXL'09 May 20, 2009

Remote Station Processing Boards Remote Station Processing Boards  FPGAs  PPF: creates 512 * 195 KHz subbands  select up to 164 subbands  beam form LBAs/tiles  UDP packets over WAN to correlator ScicomP/SP-XXL'09 May 20, 2009 16 ScicomP/SP-XXL'09 May 20, 2009

Transient Buffer Boards Transient Buffer Boards  4 sec. raw antenna data stored in TBB  trigger → freeze → dump → post analysis  not possible with dishes! ScicomP/SP-XXL'09 May 20, 2009 17 ScicomP/SP-XXL'09 May 20, 2009

Stations Stations  ≤ 2009: prototypes  building real stations now  18–25 core  18–25 remote  8–20 European  dedicated fibers to correlator ScicomP/SP-XXL'09 May 20, 2009 18 ScicomP/SP-XXL'09 May 20, 2009

Observation Characteristics Observation Characteristics  2 polarizations  32 MHz bandwidth from 1 mode  select 164 * 195 KHz subbands  up to 8 concurrent observations  trade bandwidth for beams ScicomP/SP-XXL'09 May 20, 2009 19 ScicomP/SP-XXL'09 May 20, 2009

LOFAR Processing LOFAR Processing ScicomP/SP-XXL'09 May 20, 2009 20 ScicomP/SP-XXL'09 May 20, 2009

Central Processing Pipelines Central Processing Pipelines  standard imaging mode  pulsar survey mode  known pulsar mode  transients mode  very/ultra high-energy modes  ... ScicomP/SP-XXL'09 May 20, 2009 21 ScicomP/SP-XXL'09 May 20, 2009

Blue Gene History Blue Gene History  6 racks Blue Gene/L (2005–2008)  2½ rack Blue Gene/P (2008–) ScicomP/SP-XXL'09 May 20, 2009 22 ScicomP/SP-XXL'09 May 20, 2009

The Blue Gene/P The Blue Gene/P  850 MHz PPC  4 cores * 2 FPUs * 1 FMA/cycle  complex numbers  3-D torus, collective, barrier, 10 GbE, JTAG networks  2½ racks = 10,880 cores = 37 TFLOP/s + 160*10 Gb/s ScicomP/SP-XXL'09 May 20, 2009 23 ScicomP/SP-XXL'09 May 20, 2009

BG/P Pset BG/P Pset  I/O Nodes (ION) & Compute Nodes (CN)  ION handles I/O requests of CN  transparent  ION:CN = 1:16  64 IONs/rack ScicomP/SP-XXL'09 May 20, 2009 24 ScicomP/SP-XXL'09 May 20, 2009

The BG/P Correlator The BG/P Correlator  three distributed applications/platforms  BG/P I/O nodes (ION)  BG/P compute nodes (CN)  external storage nodes ScicomP/SP-XXL'09 May 20, 2009 25 ScicomP/SP-XXL'09 May 20, 2009

Application Software on I/O Node Application Software on I/O Node  unorthodox  more efficient & flexible  BG/L: saved costs; for input cluster  BG/L: major system software changes (ZOID) (thanks ANL!) [PPoPP'08]  BG/P: better support ScicomP/SP-XXL'09 May 20, 2009 26 ScicomP/SP-XXL'09 May 20, 2009

I/O Node Processing I/O Node Processing  two sections  input  output  multi threaded ScicomP/SP-XXL'09 May 20, 2009 27 ScicomP/SP-XXL'09 May 20, 2009

I/O Node Input Section I/O Node Input Section  ION receives from 1 station  48,828 pkt/s  handles missing packets ScicomP/SP-XXL'09 May 20, 2009 28 ScicomP/SP-XXL'09 May 20, 2009

Circular Buffer Circular Buffer  circular buffer (~2.5 s)  WAN delays  delay stream  handle hiccups Δt = 22μs ≈ 4 * 5.12 μs samples ScicomP/SP-XXL'09 May 20, 2009 29 ScicomP/SP-XXL'09 May 20, 2009

→ Compute Node I/O Node → Compute Node I/O Node  ION sends data to CN  wall-clock time trigger  chunk  = 196,608 samples (1.007 s), 1 subband, 2 pols, 1 station ScicomP/SP-XXL'09 May 20, 2009 30 ScicomP/SP-XXL'09 May 20, 2009

Compute Node Processing Compute Node Processing ScicomP/SP-XXL'09 May 20, 2009 31 ScicomP/SP-XXL'09 May 20, 2009

Exchange Exchange  hundreds of Gb/s  asynchronous ScicomP/SP-XXL'09 May 20, 2009 32 ScicomP/SP-XXL'09 May 20, 2009

PolyPhase Filter PolyPhase Filter  splits subband into channels  time vs. frequency resolution  FIR filter + FFT  allows narrow-band RFI removal ScicomP/SP-XXL'09 May 20, 2009 33 ScicomP/SP-XXL'09 May 20, 2009

Phase Correction Phase Correction  correct observation direction  already shifted samples — correct rest  interpolate Δt = 22μs = 4 * 5.12 μs samples + e -2iπ f *1.52 ScicomP/SP-XXL'09 May 20, 2009 34 ScicomP/SP-XXL'09 May 20, 2009

Band Pass Correction Band Pass Correction  channel powers unequal  caused by station PPF power  correct channel ScicomP/SP-XXL'09 May 20, 2009 35 ScicomP/SP-XXL'09 May 20, 2009

Beam Forming Beam Forming  add group of stations to form “superstation”  optional ScicomP/SP-XXL'09 May 20, 2009 36 ScicomP/SP-XXL'09 May 20, 2009

Correlate Correlate  filters noise  multiply samples of all pairs of stations  integrate over time ScicomP/SP-XXL'09 May 20, 2009 37 ScicomP/SP-XXL'09 May 20, 2009

Correlator Output Correlator Output 60.1 frequency (MHz) 59.9 0 time (h) 9  correlations between two stations  color = phase, intensity = power  combined contribution of (strong) sources  earth rotation changes phase ScicomP/SP-XXL'09 May 20, 2009 38 ScicomP/SP-XXL'09 May 20, 2009

Work Distribution Work Distribution  process subbands independently  stations must be combined  chunk needs > 1 second processing time  round-robin distribution  receive, process, send, idle  OVERLY SIMPLIFIED! ScicomP/SP-XXL'09 May 20, 2009 39 ScicomP/SP-XXL'09 May 20, 2009

I/O Node Output Section I/O Node Output Section  (adds correlations)  best-effort queue  ensures real-time continuation of correlator ScicomP/SP-XXL'09 May 20, 2009 40 ScicomP/SP-XXL'09 May 20, 2009

I/O Node Real-Time Scheduling I/O Node Real-Time Scheduling  use Linux RT scheduler ScicomP/SP-XXL'09 May 20, 2009 41 ScicomP/SP-XXL'09 May 20, 2009

I/O Node Memory I/O Node Memory PPC 450: software TLB-miss handler [P2S2'09]  Linux: slows down applications by 40%−300%  modified kernel to provide 6 * 256 MiB “fast” pages (thanks ANL!)  ScicomP/SP-XXL'09 May 20, 2009 42 ScicomP/SP-XXL'09 May 20, 2009

Storage Storage  correlations saved on disk  external cluster  ~1 PB  post-processed within week ScicomP/SP-XXL'09 May 20, 2009 43 ScicomP/SP-XXL'09 May 20, 2009