Monitoring, Run Control, Data Management Geoff Savage, Wes Ketchum SBN Online Systems 16 Feb 2016
Fermilab Contributions • This presentation assumes you have read the slides from previous meetings – Not a “complete” set of slides • Topics covered here – Online Monitoring – Run Control – Data Management • DAQ/Data focused topics – Your favorite topic might not be covered in this presentation – No trigger/clock, slow controls, … • Where do we draw the line? • Look a bit deeper 2 3/9/16 Geoff Savage | SBN Online Systems
Where We Started 3 3/9/16 Geoff Savage | SBN Online Systems
Where We Started Experiment Shared Specific 4 3/9/16 Geoff Savage | SBN Online Systems
Areas of Expertise • Right away I see two areas of expertise needed with this interface selection, no sharing here • Retrieve data from hardware buffer • Access data in events and fragments • These interfaces require expert knowledge of – Data link – Detector contruction – Electronics • Could be simplified with common data formats 5 3/9/16 Geoff Savage | SBN Online Systems
Data Flow Monitor Hardware (by sub (Hit maps) system) Data Readout Electronics Acquisition Detector Node Event Event Monitor Builder Display To Offline Data Logger Disk 6 3/9/16 Geoff Savage | SBN Online Systems
Data Management • My comments are focused on the online side – See next slide for my online/offline definition – See Mike Kirby’s talk for expertise on the offline components of data management • Event building = fragments + beam data • Writing events to disk – Research on fast enough disk technologies • MicroBooNE 300+ MB/s with ”traditional” spinning drives, RAID 6 • Solid state? Not needed if more than one event builder/aggregator • Monitoring stream separate from data stream • Monitor data flow (don’t look inside events) – Graphical display – Slow control system • DAQ applications push data into a slow control system • Log files 7 3/9/16 Geoff Savage | SBN Online Systems
Online/Offline Interface Offline Online Disk Online Offline 8 3/9/16 Geoff Savage | SBN Online Systems
Run Control • Offsite access with authentication required – Web-based? VNC-based? • Configuration – Create named configurations – Identify electronics and software settings • Coordination – Download a requested configuration (by name) • Operation – Start/Stop/Pause/Resume – Auto pause • Run history in data base – Including history of run actions! • Automatic log book entries • Alarm notification – visual and audible • (Need a picture for these items) 9 3/9/16 Geoff Savage | SBN Online Systems
Online/Nearline Monitoring • Detector hardware (hitmaps) – Comparison with reference – Automated alarms possible • And checklist-like comparisons for shifters • Slow controls – EPICS? See talk from Glenn today/next week if needed • Computer hardware (Ganglia, cron scripts) • Alarms – Immediate visual and audible notification – Email – Searchable for trends database record – Integrated across system: one interface for shifters 10 3/9/16 Geoff Savage | SBN Online Systems
Other Items to Share • Naming conventions - Simplify filtering • Programming languages – C++, Python, Java, ROOT, LabView, html5, javascript , … • Network architecture – Data, Monitoring, Interactive, Trigger (White Rabbit) • Infrastructure tools – Log book auto insertion, Database I/O and display, Log files • Computer architecture – NFS server, Linux, control room, gateway, online farm, … • Disk server – need fast write speeds • Alarms • Web/GUI framework – consistent look and feel 11 3/9/16 Geoff Savage | SBN Online Systems
Other Items for Consideration • Getting to the final picture requires many steps – Don’t try to be perfect right away • Testing individual systems – Fermilab with full support – Fermilab with partial support (test stand) – Non-Fermilab locations • Integration – Operate systems individually – Operate subset of systems together • Simulation modes • Scope mode • Coding standards and reviews 12 3/9/16 Geoff Savage | SBN Online Systems
Fermilab Contributions • LarSoft • ARTDAQ • Run control • Log files – All software applications should create log files with consistent entries – Send log entries to a server and let the server handle the files – FNAL Message Facility can log/display messages at different severity • Artdaq should have nice display for this soon • Alarm handling – By experiment – Summary of alarms from all experiments? • Communication protocol • Log book entry insertion software • Coordination of projects 13 3/9/16 Geoff Savage | SBN Online Systems
Other Fermilab Contributions • Items listed in previous slide require support from other groups at Fermilab • Computer system administration – SLAM team huge asset • Networking • Database • Log book • Production/data management monitoring – OPOS team in SCD • Education – Sponsor classes teaching chosen technologies 14 3/9/16 Geoff Savage | SBN Online Systems
Recommend
More recommend
