Shifan Zuo NAOC September 19, 2018 Outline The Tianlai data - PowerPoint PPT Presentation

Overview of Tianlai’s data processing pipeline and some results Shifan Zuo NAOC September 19, 2018

Outline The Tianlai data processing pipeline — tlpipe 1 Overview Main Results 2 RFI Flagging Calibration Map-making Issues and Plans 3

tlpipe The Tianlai data processing pipeline — tlpipe https://github.com/TianlaiProject/tlpipe . a Python package specifically developed for the Tianlai array almost complete for the early stage of data processing tasks reading data from raw observing data files RFI flagging noise source calibration sky point source calibration data binning map-making data selection, transformation, visualization use HDF5 file format based on the MPI framework, scales from 1 to ∼ 10 4 MPI processes built on the Python scientific computing stack: numpy, scipy, matplotlib, h5py, etc performance-critical parts are statically compiled by using Cython 3 / 17

Three Main Components of tlpipe the Task manager the Tasks the Data container Task1 Task manager Task2 Data container Task3 The task manager executes tasks according to the settings in an input parameter file provided by the user. 4 / 17

Data Processing Pipeline Schematic of the data processing pipeline Implemented Tasks RFI flagging rfi flagging.py, line rfi.py, time flag.py, freq flag.py, multiscale flag.py, combine mask.py, sir operate.py, rfi stats.py Calibration ns cal.py, ps cal.py, apply gain.py Map-making gen beam.py, map making.py Transformation delay transform.py, rt2ts.py, temperature convert.py, phs2src.py, phs2zen.py, re order.py, freq rebin.py Visualization plot integral.py, plot slice.py, plot waterfall.py, plot phase.py Others dispatch.py, detect ns.py, bad detect.py, phase closure.py, daytime mask.py, sun mask.py, accumulate.py, barrier.py, average.py 5 / 17

Outline The Tianlai data processing pipeline — tlpipe 1 Overview Main Results 2 RFI Flagging Calibration Map-making Issues and Plans 3

RFI Flagging SumThreshold Offringa et al. 2010 The sum of a combination of one or more samples is used as a threshold criterion. χ 1 χ i = ρ log 2 i SIR operator Offringa et al. 2012 The scale-invariant rank operator will flag a subsequence when more than (1 − η ) N of its samples are flagged, with N the number of samples in the subsequence and a constant, 0 ≤ η ≤ 1 . 7 / 17

Calibration noise source calibration ij = G ij ( V sky V on + V ns ij + n ij ) ij ij = G ij ( V sky V off + n ij ) ij V on ij − V off ij = G ij V ns ij + δn ij ≈ C | G ij | e ik ∆ L ij e ik ( r i − r j ) , ∆ L ij : the equivalent cable delay φ ij = Arg ( V on ij − V off ij ) = k ∆ L ij +const ., sky point source calibration For a dominated point source V ij = V 0 ij + n ij V 0 ij = S c G i G ∗ j n 0 ) e 2 πi ˆ n 0 · u i , G i = g i A i (ˆ 8 / 17

Stable PCA Decomposition In matrix form, the observed visibility V is the sum of three terms V = V 0 + S + N , where V 0 = S c GG † , S is outliers, a sparse matrix, N is noise. This is achieved by solving 1 2 � V − V 0 − S � 2 min F + λ � S � 0 s.t. rank( V 0 ) ≤ 1 V 0 , S (a) V (b) V 0 (c) S (d) N 9 / 17

Component Separation by Stable PCA Cyg A Sun Cas A Tau A 10 / 17

Gain and Beam Profile V 0 = S c GG † n 0 ) e 2 πi ˆ n 0 · u i G i = g i A i (ˆ | G i | = | g i | A i (ˆ n 0 ) ∝ A i (ˆ n 0 ) 11 / 17

Redundant Baseline Check V before cal V after cal V 0 before cal V 0 after cal 12 / 17

Map-making m-mode map-making method Shaw et al. 2014 v α � B α lm a lm + n α m = m l v = B a + n 13 / 17

Map-making Tikhonov-regularized m-mode map-making method min � v − B a � 2 + ε � a � 2 a = ( B ∗ B + ε I ) − 1 B ∗ v ˆ 14 / 17

Deconvolution Correction A deconvolution improvement process a = ( B ∗ B + ε I ) − 1 B ∗ v ˆ and v = B a + n a + ∆ a − n ′ ⇒ a = ˆ = ( I + ∆ + · · · + ∆ n )ˆ a + ∆ n +1 a + noise term. where ∆ = ε ( B ∗ B + ε I ) − 1 , n ′ = ( B ∗ B + ε I ) − 1 B ∗ n . (a) n = 1 (b) n = 5 15 / 17

Issues and Plans More efficient I/O collective I/O, asynchronous I/O, ... Higher performance Cython, numba, ... Better RFI Flagging machine learning, deeep learning, ... Beam profile simulation, measurement, ... Better calibration flux, direction-dependent, polarization, ionospheric variation, ... Map-making and decomposition real time / instantaneous imaging, CLEAN, ... ... 16 / 17

More About the Deconvolution A deconvolution improvement process a = ( B ∗ B + ε I ) − 1 B ∗ v ˆ and v = B a + n a + ∆ a − n ′ ⇒ a = ˆ = ( I + ∆ + · · · + ∆ n )ˆ a + ∆ n +1 a − ( I + ∆ + · · · + ∆ n ) n ′ where ∆ = ε ( B ∗ B + ε I ) − 1 , n ′ = ( B ∗ B + ε I ) − 1 B ∗ n . Without noise, information that can not be recovered is ∆ n +1 a . Use the SVD of B = UΣV ∗ , we have ∆ = ε V ( Σ 2 + ε I ) − 1 V ∗ � 1 if σ i = 0 i + ε ) n = Eigenvalue of ∆ n : ( ε if σ i > 0 when n → ∞ . σ 2 0 In priciple, we can recover all that can be recovered with large enough n without noise, but B should be accurate. Also we can show that ( I + ∆ + · · · + ∆ n ) n ′ is finite when n → ∞ . 17 / 17