Plans and Policies for LSST Alert Distribution Eric Bellm Alert - PowerPoint PPT Presentation

Plans and Policies for LSST Alert Distribution Eric Bellm Alert Production Science Lead, LSST DM University of Washington August 16, 2018 #lsst2018 #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 1

LSST’s Prompt Processing system will enable real-time science. After an LSST image is taken, image subtraction will identify sources that move or vary within 60 seconds. All of these Difference Image Analysis Sources (DIASources) will be packaged with contextual information into world-public alert packets for distribution. The full stream of alerts will be sent to a small number of “community brokers”, who will add value to the stream and provide user access. LSST’s alert packets are world-public. Additionally, LSST data rights holders will have access to a limited-capability simple filtering service (the “mini-broker”). There are many open questions about this ecosystem that we are now resolving. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 2

Outline Review of the Alert Production System Overview of the LSST mini-broker Towards a selection process for community brokers #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 3

The Alert Production System #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 4

Alert Production identifies time-varying objects. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 5

Rich alert packets enable powerful filtering. Data Products Definition Document ls.st/dpdd #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 6

We are testing the performance of new alert distribution technologies. Testing a system using open source technologies: Apache Avro (binary serialization format for alerts) Apache Kafka (realtime streaming platform; used for alert distribution) LSST-scale experiments detailed in Data Management Technical Notes #28 (dmtn-028.lsst.io) & #81 (dmtn-081.lsst.io) acceptable performance at LSST scale Met NSF Level 2 milestone for Alert Distribution (LDM-503-5) in July A fork of this technology stack has recently been put into production by the Zwicky Transient Facility (ZTF) collaboration, with realtime distribution of alerts to ANTARES, Lasair (UK), ALeRCE (Chile), LCO #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 7

The LSST Mini-Broker #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 8

The LSST mini-broker provides simple filtering capabilities. Users can create filters that return (in near real-time) a subset of LSST alerts based only on data in the alert packet e.g., can use lightcurve, variability parameters, colors, etc., but no crossmatches to external catalogs still provides a powerful initial selection Runs in the LSST Data Access Center (-> users must have data rights) Will have TBD capacity limits on runtime, number of passed alerts, number of filters run #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 9

Simple filters can enable powerful searches. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 10

A range of output channels are being explored. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 11

Towards selection of community brokers #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 12

Community brokers are vital to enabling scientific utilization of the LSST alert stream. May: • Provide public access to alerts • Crossmatch to other catalogs or data streams • Classify events • Provide filtering, visualization, and search • Coordinate scientific activity and/or followup observations • Aggregate alert annotations (community classifications, etc.) • … probably more! #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 13

The number of community brokers will be finite. Outbound bandwidth from the datacenter is the expected bottleneck Early planning led to an expectation of ~4 full streams this is not firmly established; we are revisiting the issue If true capacity is not greater than the number of credible brokers, a broker selection process is needed. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 14

We are considering an “open” call for community brokers. Require demonstration of technical capability (see next slide) but no functional requirements, e.g., • No requirement to receive the full stream • No requirement to redistribute the full stream • No requirement to make products world public But evaluation criteria (subsequent slides) are expected to favor proposals that offer these! Any institution worldwide can propose Hope to enable a rich broker ecosystem offering diverse capabilities #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 15

Brokers must demonstrate adequate resources to accomplish their proposed goals. Large inbound and outbound network bandwidth (the full alert stream is a few TB/night) Petabytes of disk capacity Databases handling of billions of sources Compute resources to handle sophisticated classification and filtering tasks in real time at scale Appropriate personnel to develop and maintain the service Institutional & funding support to ensure the longevity and stability of the service. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 16

We propose brokers be evaluated on their contribution to the scientific utilization of LSST. Serve a large community Enable high-profile science Provide unique capabilities Contribute to LSST’s four science pillars Take advantage of the unique aspects of the LSST alert stream (real-time, world-public) #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 17

Suggested broker evaluation criteria have been drafted. Scientific value --- see previous slides Availability of data products and services to the world community Scientific Validity --- demonstrated on precursor surveys or data Integration with the Time-Domain Ecosystem --- e.g., followup telescopes, other services Community adoption Complementarity to other selected brokers Applicable existing agreements, if any #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 18

Proposed Timeline 2018: Plans and Policies for LSST Alert Distribution document issued 2019: Issue call for Letters of Intent Document bulk transport format and interface Set up canned alert stream with sample precursor data Invite full proposals for community brokers from selected LOIs 2020: Broker proposals due Produce commissioning alerts with ComCam with substantial latency 2021: Finalize number of full streams Selection of community brokers for early operations Produce commissioning alerts with LSSTCam with substantial latency 2022: Integrate selected brokers with the LSST Alert System Begin full LSST operations 2023+: Period reviews of broker performance, potential revisions #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 19

Summary The LSST alert stream provides a powerful real-time view of the dynamic sky, but downstream tools are required to identify objects of interest. We are drafting policies to guide selection of community brokers and will release them later this year. We are prototyping technologies for the LSST mini-broker. Community brokers using the ZTF alert stream provide a near- term opportunity to test-drive LSST time-domain science. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 20