CORRELATION IMAGING Melvyn Wright, Lynn Urry, Matt Dexter, David - PDF document

NSF, Washington, Aug 2008 CORRELATION IMAGING Melvyn Wright, Lynn Urry, Matt Dexter, David MacMahon, Oren Milgrome, William Barott, Colby Gutierrez-Kraybill, Rick Forster, Garrett Keating, Gerald Harp, Dan Werthimer, Don Backer and the CASPER group Radio Astronomy Laboratory, Digital Group 1. SUMMARY • what we have done. • what we learned. • what we plan to do. • what we hope to learn.

– 2 – 2. CURRENT DIGITAL SYSTEM STATUS distributed data processing using multiple, re-usable hardware (1) ATA FX64 correlator – 100 MHz bandwidth, 1024 channels – custom F and X board design for up to 350 antennas – innovative, custom backplane solves signal distribution – pipeline correlator solves N 2 problem – 3 more FX64 correlators being built (Urry, MacMahon, Dexter) (2) CASPER – generic hardware and commercial protocols. – reuse hardware for IF processors, beamformers, correlators. – 10 Gb switch solves signal distribution (Werthimer) – flexible routing allows hardware to be maintained and upgraded. • 42-antenna beamformer [100 MHz, + 1 × N correlator for calibration] • 32-antenna beamformer [100 MHz, + 1 × N correlator for calibration] • fly’s eye machine • pulsar processor • packetized correlator on EoR array (Backer & Bradley) (3) SETI – analogue input from beamformer into PRELUDE. – 10 Gb digital input from beamformer into SONATA.

– 3 – 3. WHAT WE LEARNED • (1) vindicate generic approach. – custom designs take 5-10 years from concept to production. • Multipurpose computing platform for radio telescopes – system design philosophy treats boards as modular DSP resource. – faster to market. – allows user and student participation. • FPGA enable wide range of radio astronomy applications. – flexible interconnect architecture allows reconfiguration for multiple applications. – programming model allows focus on application rather than hardware. • (2) beamformers calibration – 1 × N correlator OK for L-band; not for X-band – need accurate calibration to form nulls. • (3) RFI excision – detailed editing can be automated. – big time saving for users. – enables real time imaging. • (4) Data management – sustained rate if we process at ATA and only keep “good” data. ∼ 100 GB/day = 10 hours processing on strato

– 4 – Fig. 1.— Bandwidth of Radio Astronomy correlators. Future correlators are shown in red. The solid line shows Moore’s Law extrapolated from an SKA correlator for 4400 antennas with 1 GHz bandwidth. The right axis gives the data rate assuming 2000 spectral channels per GHz of bandwidth, 4 bytes per channel, a 25% overhead, and a 10s sample interval. ATA-42 ∼ 100 GB/day, ALMA ∼ 3 TB/day, SKA ∼ 1 PB/day

– 5 – 4. WHAT WE PLAN TO DO • (1) Real Time Imaging • Calibration and imaging integrated with hardware. – stream data processing – flexible programming environment. • Simultaneous imaging of multiple targets. • Calibration in real time using global model. – feedback calibration into beamformers and imagers. • Subtract global model from uv data before imaging. • Real time images update global model – model becomes final image when observations completed. • Calibrated images as normal output. • (2) narrow-band software beamformer/correlator – 64 ant × 2 pol × 20 MHz • (3) wide-band gateware beamformer/correlator – ATA correlator 64 ant × 2 pol × 600 MHz – CARMA correlator 23 ant × 2 pol × 8 GHz – VLBI beamformer 10 ant × 1 pol × 1 GHz • (4) passive radar – long delay and fast fringe rate in iBOB gateware.

– 6 – Gains( s ,f,p ) Model( s ,f,p ) PBeam( s ,f,p ) Bandpass( ) f Packetized PolCal( f,p ) � F X I 1 Solver X � � ˆ ˆ V f V ( ) p s � 1 , � X F Astronomy Ethernet�Switch I 2 Imager � � ˆ V f V ( ˆ ) p s � , 2 � F ... G Solver � � k j X I ~ k j � F P Beam � Control Fig. 2.— Data flow from telescopes to images.

– 7 – 5. WHAT WE HOPE TO LEARN • Real Time Imaging – transient astronomy – high fidelity imaging • FPGA gateware survives by using a technology independent design flow. – porting toolflow to new versions – porting gateware to new FPGA • Best use of telescope and human resources. – commensal observing • How to build and operate SKA

– 8 – Fig. 3.— 10 pointings × 8 epochs of the ATATS field. These data were flagged using Garrett Keating’s automatic editing. All 80 pointings were then mosaiced in MIRIAD to produce the image. The rms ∼ 4 mJy. Circles indicate NVSS sources down to 15 mJy, with the radius scaled with the NVSS flux density. Steve Croft 31July2008

– 9 – 6. MOTIVATION FOR REAL TIME IMAGING SCIENCE • Delayed calibration and analysis limit science. • Transient sources. • Targets of opportunity. • Observations directed to meet science goals. TECHNICAL • Non coplanar array geometry • Non isoplanicity of atmosphere. • Sources in sidelobes of beams. • Beam pattern time variable • RFI handled as data are acquired. • Off-line data reduction limits data rates. USER • Telescopes used by non experts. • Primary interest is astronomy, not data processing. • Off-line data reduction expensive and time consuming. • Expertize many astronomers do not have or want. • Best use of telescope and human resources.

– 10 – 7. SYSTEM ARCHITECTURE Goal is to reduce data rate from sky to astonomer DIGITIZE • Total data bandwidth is N × 2 B × N pol × N bits 4 10 11 ( N/ 100) ( N pol / 2) ( B/GHz ) ( N bits / 8) bytes/s CHANNELIZE • Large N , high dynamic range favors FX design. – spectral resolution – bandwidth smearing – RFI mitigation (pre- and post correlation) • Polyphase filter. – Excellent channel separation. – Low cost ∼ log ( N chan ) – Full bit growth then 4-bit selection for each channel. • Parallel data processing. – Rate in each channel by factor N chan . e.g. with N chan = 1000 2 10 8 ( N/ 100) ( N pol / 2) ( B/GHz ) ( N bits / 4) bytes/s/chan – calibration and imaging timing linear with channels and number of records. 10x as many channels == 10x as many records partitioning data over multiple processors is linear. up to disk I/O available; on strato is 600MB/s - 2GB/s

– 11 – CORRELATOR • Data bandwidth for siderial source anywhere in sky: 10 7 ( N/ 100) 2 ( N pol / 4) ( N chan / 10 3 ) ( N bytes / 5) ( D max /km ) ( λ/m ) − 1 bytes/s INTEGRATE AT MULTIPLE PHASE CENTERS • Reduce data rate • Range of fringe rates within the FoV limited by: – primary beam, – isoplanatic patch, – non coplanar baselines, ∼ sqrt ( λ/D max ). • Primary beam imaged by N f ∼ λD max /D 2 ant images using 2D FFT. e.g. ATA, with D ant = 6m, D max = 1 km, FoV defined by primary beam FWHM , ∼ 17 arcmin at λ = 3 cm, and by non coplanar baselines at λ = 1 m. • CMAC for each baseline and frequency channel, but data rate reduced. – data bandwidth for imaging primary beam FWHM : 10 6 ( N/ 100) 2 ( N pol / 4) ( N chan / 10 3 ) ( N bytes / 5) ( D max /km ) ( D ant /m ) − 1 bytes/s • REAL TIME IMAGING – 1 image per hour

– 12 – Fig. 4.— Multichannel Calibration and Imaging

– 13 – 8. CALIBRATION • Calibrate using global sky model. • Separately calibrate each isoplanatic patch. • Global calibration model across array. • Image regions of interest and confusing sources. • Stream data processing. Algorithm • Calculate model visibility from sky model ′ V j,k = exp (2 πi/λ r.s 0 ) × Σ I × A × B × P × G × exp[2 πi/λ r. ( s − s o )] – I ( s, ν, p ) sky model. – A ( s, ν, p ) primary beam, – B ( ν ) instrumental bandpass, – P ( s, ν, p ) polarization calibration, – G ( time, s 0 ) gain versus time and phase center. – r antenna baseline vector. – s , ν , and p position, frequency and polarization. – s o is the phase center for each region of interest. • Measurement equation for A, B, P , and G . – decompose into antenna dependent components G jk = g j × g k . – use a-priori measurements wherever possible. • Global sky model to determine antenna gains g ( t, s 0 ) . – gains measure tropospheric and ionospheric delays. • Self calibration. χ 2 = < Σ[ V × g i g j − V ′ ] 2 / σ 2 ij > – χ 2 accumulated in distributed processors.

– 14 – 9. IMAGING • Parallel processing in distributed architecture. • Subtract a-priori source model from calibrated uv data. – confusion and sidelobe subtraction. • Image each phase center. – image size < D max /λ for 2D FFT, ∼ 10 8 for 1000 km baseline at λ 1 m. • Deconvolution minimized by good uv coverage for large N . • Imaging guided in real time by convergence of model and χ 2 image. – variable sources are inconsistent with the global model. – χ 2 image identifies transient sources. • Phase centers can be moved to image regions for science goals, – image confusing sources. – image different isoplanatic patches. • Images update global model. • Model and calibration improve as observations proceed. – observe until model consistent with uv data streams.