Streaming Realtime Workflows at the Light Sources Harinarayan Krishnan, Computer Systems Engineer Computational Research Division, Data Analysis & Visualization Group
What is a workflow?
A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic organization of resources into processes that transform materials, provide services, or process information.
Post processing Workflows Simulation http://www.gridprovenance.org/applications/DLR.html Page
Insitu Workflows In-situ workflows Realtime Steering https://www.researchgate.net/publication/320237199_Development_of_Advanced_Analysis_Toolkit_for_Turbulent_Bubbly_Flow_Sim ulations/figures?lo=1 Page
Realtime Workflows https://www.semanticscholar.org/paper/Improved-%24%24T_%7B2%7D%5E%7B*%7D%24%24-T-2-% E2%88%97-determination-in-23Na%2C-Niesporek-Umathum/c16507726ba126426a58db5172aa222a57b50e13/figure/2 Page
Bright and coherent X-rays ALS, Berkeley, US Sprint8/SACLA, Japan Max IV, Lund, Sweden …and many more SSRL/LCLS, Stanford, US PSI, Switzerland Elettra/FERMI, Trieste, Italy Page 7
Data Acquisition Page
Data Rates Current Data Rates: 400 megabytes per second and can now generate a few terabytes of data per day – enough to store about 500 to 1,000 feature-length movies. Next-generation detectors, will produce data 100 times faster Page
Higher Contrast & Time Dependence Future DAQs are also higher contrast than current ALS beamlines. This brightness can translate to nanoscale resolution, and can also enable far more precision in time-dependent experiments Page
“ Under the ideal situation, where we have very high- contrast samples, we’ll be able to image at the x-ray wavelength, which nobody else can do. COSMIC is going to bring x-ray microscopy much closer to the capabilities of electron microscopy, but with the added benefit of x-rays, which is that you can penetrate lots of material. ” ~ David Shapiro (COSMIC, ALS) Page
Why Streaming? Complexity • Workflows • Data analysis • Resource management • Scalability and Portability Resources • Experiment Timeframe is limited • Beam time is not free We start with a use case: Ptychography is one of the most data intensive beamlines. Page 12
What is Ptychography? Page
Ptychography is similar to Scanning Microscope but trades greater complexity for higher resolution 40 horizontal 1600 diffraction images positions 260 40 vertical positions 260 Making use of redundant data! Page 14
Ptychography Pipeline Scanned Sample Diffraction Pattern Beamline Ptychography Zone Plate Lens Frame Stack X-ray Beam Final CCD Output Scan Detector Direction For each pixel replace magnitude Multiply Object Iteration i with experimental value with Probes FFT Overlap and Algorithm Inverse FFT average frames. Page 15
SHARP: Scalable Heterogeneous Adaptive Real-time Ptychography Ptychography image using the same data. Traditional STXM SEM image. image . Resolution of about 10 nm. CAMERA developed SHARP: A collection of algorithms packaged as useable software for Ptychographic reconstruction • Developed New Accelerated Solvers (RAAR and ADMM) – MPI & Multi-GPU • Combined Phase Retrieval and Denoising • Proving Convergence and Stability with First Order ADMM RAAR: Relaxed Averaged Alternating Reflections ADMM: Alternating Direction of Method of Multipliers Page 16
Why is this important? US researchers claim X-ray microscope record US researchers have used soft X-ray Ptychography to image structures at 5 nm scale . The resolution, obtained at Berkeley Lab's Advanced Light Source, is the highest resolution ever achieved with X-ray microscopy https://microscopy-analysis.com/editorials/editorial-listings/us-researchers-claim-x-ray-microscopy-record Page
Ptychography Workflow 5 UDP Trigger 4 1 2 3 Experiment Framegrabber / Reconstruction Pre-processing Control Camera control TIFF CXI file metadata Page 18
Ptychography Time Budget 5 UDP Trigger 4 1 2 3 Experiment Framegrabber / Reconstruction Pre-processing Control Camera control ~Minutes 1 frame / second time / scan (detection AND exposure): 300 (s) • Beamlines are exploratory and is run until sample or time / sample setup (s) 600 feature is found. samples or time points / user 25 • In a post processing workflow: Preprocessing & time/ user setup (s) 1200 Reconstruction can only run after all frames have been Total: 1200 + (600*25) 4.5 Hours! acquired! Page 19
Ideal Streaming Workflow 5 UDP Trigger 4 1 2 3 SHARP Experiment Framegrabber / Pre-processing reconstruction Control Camera control TIFF TIFF TIFF Page 20
Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output i CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration i with Probes FFT Overlap and lFFT average frames. Page 21
Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output j CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration j with Probes FFT Overlap and lFFT average frames. Page 22
Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output k CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration k with Probes FFT Overlap and lFFT average frames. Page 23
Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Final CCD Output Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration n with Probes FFT Overlap and lFFT average frames. Page 24
How can we build a general real-time streaming pipeline?
Goals What • Data movement • Data Tagging/Cataloging/Querying • Analysis & Visualization • Data Access When • Pre-planning – Simulation/Modeling • During Experiment – Quick Data Movement, Simple analysis, or iterative analysis • Post-planning – Archive & Compute: Typical HPC analysis Page 26
Requirements • An infrastructure to create and run complex analytical pipelines • Ability to analyze and evaluate algorithms • Get results (or partial results) for real-time decision making Page 27
Requirements User Compute Facilities Resources Performance Algorithms • Scalable • Tomography • Portable • Ptychography • Production-Ready • GiSAXS • Image Processing Page 28
CAM-Link What it is: A distributed task execution library • Interface-based: Enabling different workflows to run underneath • Coordinates communication over a distributed set of resources • Provides tasks with information and consistent environment • Handles security to communicate with services behind firewalls and batch systems. Enables developers to create and run custom workflow environments Page 29
Data Processing Pipeline Page 30
Real-time Analysis Task N Task 2 Execution steps Task 1 1.Identify and setup resources 2.Launch services Handler / event loop 3.Connect network 4.Execute graph User / local computer Compute cluster Experiment / data acquisition Page 31
Real-time Analysis Experiment / data acquisition remote event loop Compute cluster remote event loop Execution steps 1.Identify and setup resources master event loop 2.Launch services 3.Connect network 4.Execute graph User / local computer Page 32
Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P remote frame event loop grabber Control Network exp. contr ol elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute graph User / local computer Page 33
Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P remote frame event loop grabber Data Network exp. contr ol elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute graph User / local computer Page 34
Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P 10G Network remote frame event loop grabber Visualize Results exp. contr ol Write Output elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute Graph User / local computer Page 35
Real-time Analysis Page 36
Streaming workflows are necessary for many other applications
Recommend
More recommend
