SLIDE 1
Shifan Zuo NAOC September 19, 2018 Outline The Tianlai data - - PowerPoint PPT Presentation
Shifan Zuo NAOC September 19, 2018 Outline The Tianlai data - - PowerPoint PPT Presentation
Overview of Tianlais data processing pipeline and some results Shifan Zuo NAOC September 19, 2018 Outline The Tianlai data processing pipeline tlpipe 1 Overview Main Results 2 RFI Flagging Calibration Map-making Issues and Plans
SLIDE 2
SLIDE 3
tlpipe
The Tianlai data processing pipeline — tlpipe https://github.com/TianlaiProject/tlpipe. a Python package specifically developed for the Tianlai array almost complete for the early stage of data processing tasks
reading data from raw observing data files RFI flagging noise source calibration sky point source calibration data binning map-making data selection, transformation, visualization
use HDF5 file format based on the MPI framework, scales from 1 to ∼ 104 MPI processes built on the Python scientific computing stack: numpy, scipy, matplotlib, h5py, etc performance-critical parts are statically compiled by using Cython
3 / 17
SLIDE 4
Three Main Components of tlpipe
the Task manager the Tasks the Data container
Task manager Task1 Task2 Task3 Data container
The task manager executes tasks according to the settings in an input parameter file provided by the user.
4 / 17
SLIDE 5
Data Processing Pipeline
Schematic of the data processing pipeline Implemented Tasks
RFI flagging
rfi flagging.py, line rfi.py, time flag.py, freq flag.py, multiscale flag.py, combine mask.py, sir operate.py, rfi stats.py
Calibration
ns cal.py, ps cal.py, apply gain.py
Map-making
gen beam.py, map making.py
Transformation
delay transform.py, rt2ts.py, temperature convert.py, phs2src.py, phs2zen.py, re order.py, freq rebin.py
Visualization
plot integral.py, plot slice.py, plot waterfall.py, plot phase.py
Others
dispatch.py, detect ns.py, bad detect.py, phase closure.py, daytime mask.py, sun mask.py, accumulate.py, barrier.py, average.py 5 / 17
SLIDE 6
Outline
1
The Tianlai data processing pipeline — tlpipe Overview
2
Main Results RFI Flagging Calibration Map-making
3
Issues and Plans
SLIDE 7
RFI Flagging
SumThreshold
Offringa et al. 2010
The sum of a combination of
- ne or more samples is used as
a threshold criterion. χi = χ1 ρlog2 i SIR operator
Offringa et al. 2012
The scale-invariant rank
- perator will flag a subsequence
when more than (1 − η)N of its samples are flagged, with N the number of samples in the subsequence and a constant, 0 ≤ η ≤ 1.
7 / 17
SLIDE 8
Calibration
noise source calibration V on
ij = Gij(V sky ij
+ V ns
ij + nij)
V off
ij = Gij(V sky ij
+ nij) V on
ij − V off ij = GijV ns ij + δnij
≈ C|Gij|eik∆Lijeik(ri−rj), ∆Lij: the equivalent cable delay φij = Arg(V on
ij −V off ij ) = k∆Lij+const.,
sky point source calibration For a dominated point source Vij = V 0
ij + nij
V 0
ij = Sc GiG∗ j
Gi = giAi(ˆ n0)e2πiˆ
n0·ui,
8 / 17
SLIDE 9
Stable PCA Decomposition
In matrix form, the observed visibility V is the sum of three terms V = V0 + S + N, where V0 = Sc GG†, S is outliers, a sparse matrix, N is noise. This is achieved by solving min
V0,S
1 2V − V0 − S2
F + λS0
s.t. rank(V0) ≤ 1 (a) V (b) V0 (c) S (d) N
9 / 17
SLIDE 10
Component Separation by Stable PCA
Cyg A Cas A Tau A Sun
10 / 17
SLIDE 11
Gain and Beam Profile
V0 = Sc GG† Gi = giAi(ˆ n0)e2πiˆ
n0·ui
|Gi| = |gi| Ai(ˆ n0) ∝ Ai(ˆ n0)
11 / 17
SLIDE 12
Redundant Baseline Check
V before cal V after cal V0 before cal V0 after cal
12 / 17
SLIDE 13
Map-making
m-mode map-making method
Shaw et al. 2014
vα
m =
- l
Bα
lmalm + nα m
v = Ba + n
13 / 17
SLIDE 14
Map-making
Tikhonov-regularized m-mode map-making method min v − Ba2 + εa2 ˆ a = (B∗B + εI)−1B∗v
14 / 17
SLIDE 15
Deconvolution Correction
A deconvolution improvement process ˆ a = (B∗B + εI)−1B∗v and v = Ba + n ⇒ a = ˆ a + ∆a − n′ = (I + ∆ + · · · + ∆n)ˆ a + ∆n+1a + noise term. where ∆ = ε(B∗B + εI)−1, n′ = (B∗B + εI)−1B∗n.
(a) n = 1 (b) n = 5
15 / 17
SLIDE 16
Issues and Plans
More efficient I/O collective I/O, asynchronous I/O, ... Higher performance Cython, numba, ... Better RFI Flagging machine learning, deeep learning, ... Beam profile simulation, measurement, ... Better calibration flux, direction-dependent, polarization, ionospheric variation, ... Map-making and decomposition real time / instantaneous imaging, CLEAN, ... ...
16 / 17
SLIDE 17
More About the Deconvolution
A deconvolution improvement process ˆ a = (B∗B + εI)−1B∗v and v = Ba + n ⇒ a = ˆ a + ∆a − n′ = (I + ∆ + · · · + ∆n)ˆ a + ∆n+1a − (I + ∆ + · · · + ∆n)n′ where ∆ = ε(B∗B + εI)−1, n′ = (B∗B + εI)−1B∗n. Without noise, information that can not be recovered is ∆n+1a. Use the SVD of B = UΣV∗, we have ∆ = εV(Σ2 + εI)−1V∗ Eigenvalue of ∆n: (
ε σ2
i +ε)n =
- 1
if σi = 0 if σi > 0 when n → ∞. In priciple, we can recover all that can be recovered with large enough n without noise, but B should be accurate. Also we can show that (I + ∆ + · · · + ∆n)n′ is finite when n → ∞.
17 / 17