Connecting the Follow-up Chain: TOMs and Broker Requirements Rachel Street, Federica Bianco TVS Co-chairs Melissa Graham TVS member and LSST liaison R.A. Street, M. Bowman, E.S. Saunders, T. Borosona, 2018 https://arxiv.org/pdf/1806.09557.pdf
Target & Observation Discovery engines Broker services Observing facilities Manager Systems Astronomy’s Discovery Chain
Software-enabled observing programs everything we know about the LSST changing sky is in the LSST alert streams, but it is a responsiblity of the downstream user to identify the targets of interest E.Bellm, LSST 08/16/2018 Target & Observation Broker services Manager Systems TOMs
Software-enabled observing programs Programmatically requesting observations enables a lot of science, especially: • Rapid reaction science • Programs with a lot of targets • Programs with lots of observations But at high data rates, keeping track of targets, observations and data can be a big workload TOMs
TOMs: Target and Observation Manager systems • Storing target and observation information in a database o ff ers many advantages • Can be combined with DB-driven software tools to perform common tasks • Powerful user interfaces …but modern surveys issue alerts of new discoveries within minutes. We need to harvest targets of interest easily and often and quickly TOMs
Linking TOMs and Brokers To handle the data rate, TOMs need to programmatically interact with brokers …probably several brokers, ideally with a common language Once targets are harvested, astronomer’s own system/team can decide what to observe TOMs
Problems that need solving that TOMs can solve • automate alert ingestion for desired streams, ingesting multiple brokers+catalogs • automate target selection and follow up trigger based on user priorities • collect data from follow-up and organize it for further analysis and follow-up • enables selective data sharing
Examples of TOM Systems Supernova Exchange Many TOM systems share common functions, regardless of their science focus… TOMs Credit: Global Supernova Project
Examples of TOM Systems Microlensing TOM …yet many aspects are obviously tailored to the scientific goals of the project TOMs Credit: RoboNet Microlensing Team
Examples of TOM Systems NEO Exchange Building a TOM is a lot of work though TOMs Credit: LCO Solar System Team
The TOM Toolkit Project • A general-purpose software toolkit • Enable astronomers to easily build TOM systems that they can customize to suit their needs • Provide a professionally-developed codebase that will ensure that the systems are capable of scaling to future programs. Software under development now by LCO’s in-house software engineers First version expected ~mid-2019 Alpha testers will be needed! TOMs
The TOM Toolkit Project TOMs
The TOM Toolkit Project e.g.Transient Name Server, Astronomers Telegrams, the Minor Planet Center… LCO Network. Gemini, sending requests and receving status updates TOMs
The TOM Toolkit Project TOMs
TVS Broker Requirements Task Force Stimulate scientists from all fields of astronomy to think through how they will extract targets of interest from LSST, what information + data products they need at each stage, timescales of delivery and modes of interaction with brokers. Derive and document Scientific and Functional & Performance requirements Chair: R.A. Street Spokesperson: Markus Rabus Francisco Förster Buron, Suvi Gezari, Melissa Graham, Ashish Mahabal, Gautham Narayan, Keivan Stassun, Paula Szkody, Stephen Smartt, Ken Smith https://lsst-tvssc.github.io/broker_task_force_work_plan.html Broker Survey
Broker Requirements Surveys User Requirements Survey Broker Developer’s Survey 17 questions on: 19 questions on: Nature of the research Time to readiness Target selection criteria Limitations on users Mechanism to receive alerts Software requirements for users Information content of alerts Data provided Cross-matched catalogs/other surveys Functionality of interface Interface requirements: Time to access alerts • Time to access alerts Search capabilities of interface • Programmatic vs. human-interactive Interactions with other brokers • Search functions Foreseen technical challenges • Search metrics/criteria Classification functions Broker Survey (allow more descriptive answers)
Broker Survey Responses Surveys composed as Google Forms Widely circulated within the LSST Science Collaboration and Publicized outside it (open to everyone). Users survey: 61 responses Developers survey: 6 responses Results summarized in an informal report [living document]: https://github.com/LSST-TVSSC/broker-requirements-survey Broker’s Clients Survey
Broker Clients’ Questionnaire User Community Represented What is the subject area of your primary research? • Scientific diversity Broker’s Clients Survey
Broker Clients’ Questionnaire User Community Represented • Scientific diversity • Geographical distribution Broker’s Clients Survey
Broker Clients’ Questionnaire User Community Represented • Scientific diversity • Scientific diversity • Geographical distribution • Geographical distribution • Almost all already involved with LSST Do you have LSST data-access rights? Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Alert Information Content Will the LSST alert contain enough information for your purposes? Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Alert Information Content Will the LSST alert contain enough If not, what do you need? information for your purposes? Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Alert Information Content What catalogs should alerts be cross-matched against? (Essentially everything!) + 20 individual suggestions Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Speed & frequency of access to alerts How quickly should alerts be disseminated? [Multiple answers/science drivers allowed] How often would you query a broker to search for new targets? Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Target Selection Criteria 32 distinct requests for target selection filters users would like to use to identify targets of interest to them. Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Interface Functionality How do you prefer to access a broker? [Multiple options allowed] Broker’s Clients Survey
Broker Clients’ Questionnaire Result: Interface Functionality How do you prefer to access a broker? [Multiple options allowed] How would you prefer to receive alerts matching your criteria? Broker’s Clients Survey
Broker Developers’ Questionnaire Questions/answers were more descriptive. The goal was to provide a description of the status and intent of current development teams Note: 1 respondent focused on classifier algorithms rather than full-scale broker service. Broker Survey
Broker Developers’ Responses Broker Operational Goals • At least 3 brokers already have working software (ALeRCE, ANTARES, Lasair) • Several are already ingesting ZTF alerts as a pathfinder; Lasair is ingesting Pan-STARSS and ATLAS alerts • Lasair has a public interface operational now: http://lasair.roe.ac.uk/candlist/ • ANTARES expects to have a limited-feature public interface available Fall 2018 • All aim to be fully operation for the start of the LSST main survey Broker Survey
Broker Developers’ Responses Broker interface What interfaces do you plan to use to communicate with your users? Broker Survey
Broker Developers’ Responses Broker Added-Value Services All brokers plan to ingest alerts in close to realtime (within minutes) All expect to be publicly accessible, though Lasair may give access to a restricted set of value-added data products Value-Added Products include: • The classification and anomaly score of objects based on their light curves • Unique identifier, labels, timestamp, magnitudes, errors, and event classification • User-customizable output VOEvent stream. • ANTARES : LSST Alert package, computed light curve features, external catalog associations, classification and score • Lasair : Not yet decided. • ALeRCE : To be finalized, will include: ID, class probabilities and uncertainties , position, time and last magnitude and error, or light curve for certain classes; known IDs in external databases. Broker Survey
Broker Developers’ Responses Selecting targets of interest In general, brokers expect users to request all targets within a class , rather than to submit custom queries. ANTARES expects to provide limited support for user-defined filters through authenticated interfaces Several expects to issue push notifications for alerts of interest Fully user-customized filters require users to send the query to the broker in advance and receive alerts whenever a new object matches that filter. This requires authenticated user accounts. Broker Survey
Broker Developers’ Responses Broker Concerns • Complexity of machine learning algorithms and scaling their computational requirements to match the LSST alert rate • Cost of computing resources / maintaining funding • Maintaining the integrity of the database in the event of data processing mistakes or revisions • Data provenance & duplication • Early alert classification Broker Survey
Recommend
More recommend
