JGW-G1910278 KAGRA Data Tier Nobuyuki Kanda (Graduate School pf Science / NITEP , Osaka City U.) on behalf of the KAGRA collaboration at TAUP2019 8-14 September 2019 Toyama International Conference Cente � 1
2. Distribution START mirror or partial set of data 1. Store KAGRA data Collaboration Users 3. Mutual exchange of data between global GW GOAL detectors, i.e. LIGO, Virgo KAGRA Data Tier � 2
Over View Event Alert to follow-ups / counterparts Counterparts/ Follow-ups Oversea GW Data Sharing Oversea GW experiments with Other alert experiments (low latency h(t)) obs. data (bulk of data) base low latency h(t) less amount ↔ larger amount raw data raw data ~20MB/s Academia ~20MB/s Kashiwa Tier-1 Detector SINICA (Taiwan) Kamioka Tier-0 Tier-0 KISTI (Korea) (in preparation) (tunnel) + archive raw + Proc. Tier-1 archive Proc. data data ~1MB/s low latency h(t) Proc. data + partial raw data Osaka City U. RESCEU Tier-0.5 for low Tier-0.5 Nagaoka Tech. Tier-2 Proc. data ~1MB/s latency (option : raw data Niigata U, without permanent store) in KAGRA end user sites socket (KAGRA DMG software) partial GRID raw&proc. Tier-3 GRID or alternative … data set low latency (h(t)) Alart in GCN format faster (upstream) ↔ later (downstream) KAGRA Data Tier � 3
monitors Kamioka Calibration sharing environmental DGS system Storage International GW network full, science, trend proc Interferometer KAGRA Transfer, tunnel Kamioka surface building ICRR, U Tokyo (Kashiwa) LIGO Virgo system KAGRA data “frame” is GW experiments’ common format, that is packing time-series data. raw data (file): full : full data, 32sec (will be 64sec ?) “frame format” full data is essentially ’raw’ data science : subset of full data trend : re-merged as long trend data for slow sampling datas, e.g. temperature monitors. Calibrated data (channel) : Main interferometer signal will be reconstructed as time series metric strain h(t) , that are named as “ (channel definition)_C## ” C0# : level 0 = primary level, generated from the interferometer control C1# : level 1 = low latency on-line generated, using calibration signal response and impulse response filters C2# : level 2 = off-line calculated with various/fine corrections as like long-time drift. It will be released later. Proc. data (file) : Lv.0 : C00 + data quality flags Lv.1 : C00 + C1# Lv.2 : C00 + C1# + C2# KAGRA Data Tier � 4
⇩ ⇩ History of Bulk Data Transfer iKAGRA bKAGRA phase-I 3500 # of files / 100000 sec 3000 2500 2000 1500 1000 500 0 2016/07/01 2017/01/01 2017/07/01 2018/01/01 2018/07/01 2019/01/01 2019/07/01 date current bKAGRA phase-I iKAGRA iKAGRA rate cumulative bKAGRA phase-I (March 2019) (two servers) (two servers) data (2016/3/15-31, (by March 2019) (2018/4/28-5/6) 4/11-25) amount ~50 GB/hour ~54 GB/hour ~20 GB/hour raw ~450 TB ~6 TB ~3.5 TB files ~225 files/hour ~450 files/hour ~450 files/hour proc. ~1.8 TB ~ 996 GB 763 GB (full+science) Main storage (2.5PB) will be able to cover O3. We are preparing to add newer system at the spring of 2021. KAGRA Data Tier � 5
Bulk Data Rate / Amount 25 2.5 Data Rate 'full' + 'science' + 'trend' 'full 20 2.0 Cumulative Amount of Data Cumulative Amount od Data [PB] 'full' + 'science' + 'trend' ~16MB/s 'full' expected with 20MB/s Data Rate [MB/s] 15 1.5 10 1.0 ~760TiB 5 0.5 0 0.0 2016/01/01 2017/01/01 2018/01/01 2019/01/01 2020/01/01 2021/01/01 Date KAGRA Data Tier � 6
3~10 sec ~day for LIGO/Virgo) <1sec 32sec frame Bulk a few day / week low latency h(t) 1sec frame ~3 sec (9 ~15 sec KAGRA Data Tiers latency KAGRA data tree (unfinished) KAGRA site (tunnel) latency : fast Tier-0 (primary) Analysis building (Hokubu-kaikan) Tier-0.5 (low latency ICRR Kashiwa LIGO-Virgo analysis) Osaka C.U. Tier-1 Academia SINICA KISTI (*) in preparation (full data mirror) Tier-2 latency : slow NCU somewhere Nagaoka RESCEU Niigata (analysis, in Korea Tech. development) somewhere somewhere somewhere Tier-3 in Taiwan in Korea in Japan (development) (*) KISTI is in preparation for Tier-1 currently. KAGRA Data Tier � 7
Tier-0 : Main Data System at ICRR, U. Tokyo, Kashiwa 2.4 PiB (HHD) for observational data storage gpfs file system 12.8 TFLOPS since March 2017 Storage: DDN SFA7700X + SS8460 Servers: HP ProLiant DL180 G9 HP ProLiant DL20 Gen9 HP ProLiant XL170 Gen9 Internal network: Infiniband FDR KAGRA Data Tier � 8
Tier-0.5 : Osaka City Univ. Tier-0.5 is receiving low latency data, and bulk data partially. OCU’s ‘ORION’ cluster consists of 920 cores (will be 1160 at the end of September 2019) 324 TB storage Scientific Linux 7.5 HT condor KAGRA Data Tier � 9
Approximately 20 MB/s. Tier-1: Academia SINICA Tier-1 is mirror site to keep all raw data (+some.) Academia SINICA, Taiwan ~20MB/s Transfer every 100000 sec (a) recent (b) longer history Fig: Recent Tier-1 transfer rate to Academia SINICA by S.Haino Also, KISTI (Korea) is in preparation for Tier-1 now. KAGRA Data Tier � 10
主データ装置 セキュリティ上の注意は必要だが、 ルーティングを司るのは だけでよく、シンプル。 難点： セキュリティ上の注意が必要 の外で通過する機器を少なくできる（ （１）に比較すれば安全 の内部なので案 の接続は と 柏キャンパス 利点： 北部会館 改変案（２） 改変案（１） 直接のルーティングを行わさせる。 への から 一部のルーティングの改変の必要性 に、内部 坑内 ルーター 図中 柏キャンパスにおける 利点： への直接のルーティングを行わさせる。 から に、内部 柏キャンパス のそれぞれの設定はどうなるのか？ ルーティングを行うのに、 難点： で繋げられる） パス内の を出た後は、柏キャン 坑内から低遅延で観測信号を、柏キャンパスにおける サーバへ送信する。 この際、 坑内に置くキャリブレーション用の新サーバから直接に へ送信で きることが必要になる。 現状では， はプライベートアドレスを持つ サーバである。 ただし， の により，外部ネットワークから 特定のグローバルアドレスでアクセ ス可能。 神岡（北部会館）における ルーター 図中 データシステム Low Latency Data derived from CAL (Calibration Subsystem) From CAL’s strain calibrated data h(t) : A software developed by i.e. C10 level calibrated data. LIGO. Socket connection to push/pull the data. 1sec frame transfer using ‘framelink’ The software on the OS is sending direct to Kashiwa system, but the physical network path is bit complicated. The latency is ~3 sec including impulse filter calculation, format etc. iKAGRA data system KAGRA 坑内 柏キャンパス :iKAGRA データシステム KAGRA 北部会館 hyades-01 KAGRA main system perseus-01 aldebaran 柏キャンパス :KAGRA 主データ装置 VPN2 HUB3 VPN hyades-02 HUB1 HUB2 perseus-02 Infiniband andromeda-01 storage pegasus-01 VPN1 new sever CAL’s real time h(t) FW SINET KAGRA Hokubu-Kaikan ICRR Kashiwa campus tunnel (surface building) KAGRA Data Tier � 11
Toss the data to analysis pipelines Low latency data have to toss the data analysis pipelines. We connect (and are preparing) -‘framelink’ shared memory - dump to files LIGO, Virgo Kashiwa (ICRR) Kashiwa Kashiwa low latency h(t) by ‘framelink’ dedicated server for data sharing main system shared main storage files Calibration frame memory (1sec buffering) server merge (in tunnel) (working for installation) ‘framelink’ CBC Burst (plan) pipeline pipeline cluster cluster ‘framelink’ Osaka City U. RESCEU Tier-0.5 for low CBC latency KAGRA Data Tier � 12
Data exchange between LIGO,Virgo and KAGRA Low Latency h(t) : 1sec frame with a KAGRA LLO LHO Virgo tunnel, latency of second order Kamioka Exchange (sharing) is connected KAGRA Kashiwa and is working. main system ( Thanks for LIGO/Virgo great cooperation!) ….(many systems)…. main with ‘framelink’ system (2.5 PB storage) Caltech Latencies are depend on network data server for condition. sharing Server low latency LHO, LLO -> Kashiwa : 6~14 sec shared shared framelink Virgo -> Kashiwa : 10~16 sec memory memory push/pull KAGRA tunnel -> Kashiwa : ~3 sec Kashiwa (Note : These latency include h(t) reconstruction calculation.) CBC burst Bulk data exchange is also in preparation. Osaka C.U. KAGRA Data Tier � 13
Recommend
More recommend
