Experience with Model Predictive Control and Model-Based - PowerPoint PPT Presentation

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Experience with Model Predictive Control and Model-Based Reinforcement Learning Auralee Edelen Mar. 1 2018, ICFA Workshop on ML for Particle Accelerators Work with Sandra Biedron, Daniel Bowring, Brian Chase, David Douglas, Jonathan Edelen, Chip Edstrom, Denise Finstrom, Henry Freund, Stephen Milton, Dennis Nicklaus, Jinhao Ruan, Jim Steimel, Chris Tennant, Peter van der Slot, and many others

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC The Landscape of this T alk… Classic Model Online: Predictive Control Virtual Diagnostics (nominally good for systems Individual + fast execution with “long” time Components/ Use with Control + dependencies relative to Sub-systems Model combine a prioi control interval) Learning and empirical Offline: Higher-level results Controller Development Accelerator (i.e. system model) Pre-train or Update + Models NN policies Machine Optimization Encode Existing Static Policy Deployment Policy - If adaptive ML policy for tuning: gain some of the Learning same advantages as using direct online optimization Adaptive Learn Policy from + remember previous solutions / interpolate Deployment (state à action) Scratch (useful if drift is small?)

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Online Modeling • Use a machine model during operation • • Ideally: • Fast-executing, but accurate enough to be useful • Use measured inputs directly from machine • Combine a priori knowledge + learned parameters • Applications: • A tool for operators + virtual diagnostic • Predictive control • Help flag aberrant behavior • Bonus: control system development

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Online Modeling One approach: faster modeling codes Simpler models (tradeoff with accuracy) • Use a machine model during operation analytic calculations e. g. J. Galambos, et al., HPPA5, 2007 • Parallelization and GPU-acceleration of existing codes • Ideally: X. Pang, PAC13, MOPMA13 PARMILA à HPSim elegant I. V. Pogorelov, et al., IPAC15, MOPMA035 • Fast-executing, but accurate enough to be useful • Use measured inputs directly from machine Improvements in underlying modeling algorithms • Combine a priori knowledge + learned parameters • Applications: • A tool for operators + virtual diagnostic • Predictive control • Help flag aberrant behavior • Bonus: control system development

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Online Modeling One approach: faster modeling codes Simpler models (tradeoff with accuracy) • Use a machine model during operation analytic calculations e. g. J. Galambos, et al., HPPA5, 2007 • Parallelization and GPU-acceleration of existing codes • Ideally: X. Pang, PAC13, MOPMA13 PARMILA à HPSim elegant I. V. Pogorelov, et al., IPAC15, MOPMA035 • Fast-executing, but accurate enough to be useful • Use measured inputs directly from machine Improvements in underlying modeling algorithms • Combine a priori knowledge + learned parameters Another approach: machine learning model • Applications: • A tool for operators + virtual diagnostic Once trained, neural networks can execute quickly • Predictive control Train on results from slow, high-fidelity simulations • Help flag aberrant behavior Train on measured results • Bonus: control system development

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Online Modeling One approach: faster modeling codes Simpler models (tradeoff with accuracy) • Use a machine model during operation analytic calculations e. g. J. Galambos, et al., HPPA5, 2007 • Parallelization and GPU-acceleration of existing codes • Ideally: X. Pang, PAC13, MOPMA13 PARMILA à HPSim elegant I. V. Pogorelov, et al., IPAC15, MOPMA035 • Fast-executing, but accurate enough to be useful • Use measured inputs directly from machine Improvements in underlying modeling algorithms • Combine a priori knowledge + learned parameters (fractions of a second) Another approach: machine learning model • Applications: • A tool for operators + virtual diagnostic Once trained, neural networks can execute quickly • Predictive control Train on results from slow, high-fidelity simulations • Help flag aberrant behavior Train on measured results • Bonus: control system development Yields a fast-executing model that can be used operationally, but approximates behavior from slower, high-fidelity simulations (e.g. PIC codes, plasma acc., space charge)

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Online Modeling One approach: faster modeling codes Simpler models (tradeoff with accuracy) • Use a machine model during operation analytic calculations e. g. J. Galambos, et al., HPPA5, 2007 • Parallelization and GPU-acceleration of existing codes • Ideally: X. Pang, PAC13, MOPMA13 PARMILA à HPSim elegant I. V. Pogorelov, et al., IPAC15, MOPMA035 • Fast-executing, but accurate enough to be useful • Use measured inputs directly from machine Improvements in underlying modeling algorithms • Combine a priori knowledge + learned parameters (fractions of a second) Another approach: machine learning model • Applications: • A tool for operators + virtual diagnostic Once trained, neural networks can execute quickly • Predictive control Train on results from slow, high-fidelity simulations • Help flag aberrant behavior Train on measured results • Bonus: control system development Yields a fast-executing model that can be used operationally, but approximates An initial study at Fermilab: behavior from slower, high-fidelity simulations A. L. Edelen, et al. NAPAC16, TUPOA51 (e.g. PIC codes, plasma acc., space charge) One PARMELA run with 2-D space charge: ~ 20 minutes Neural network model: ~ a millisecond

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Model Predictive Control (Prediction + Planning) Basic concept: 1. Use a predictive model to assess the outcome of possible future actions 2. Choose the best series of actions 3. Execute the first action 4. Gather next time step of data 5. Repeat

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Model Predictive Control (Prediction + Planning) N m previous measurements Measured Variables Reference Trajectory u m (k – 1)… u m (k – N m ) y r (k)… y r (k + N p ) N p future time steps predicted Optimization of Controlled Variable Trajectories N c future time steps controlled Predicted Outputs y p (k)… y p (k + N p ) ! ! ! ! ! ! ! ! ! ! + ! − ! ! ! + ! Cost Function ! ! ! Plant Model (output variable targets) Constraints Solver ! ! ! ! ! ! !" Future Inputs ! ! , ! ! ! ! + ! − ! ! , !"# ! + ! ! ! ! ! ! ! ! u cv (k)… u cv (k + N c – 1) (controllable variable targets) ! ! ! ! ! ! !" ! ∆ ! , ! ! ! ! + ! − ! ! ! + ! − 1 u cv (k) ! ! ! ! ! ! (movement size) Plant

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Neural Network Policies and Reinforcement Learning Actor-only Methods • Actor is a control policy • Maps states to actions Reward provides training signal • Actor-Critic Methods Can train on models first to get a good initial solution before deployment Critic maps states or state/action pairs to • an estimate of long-term reward • Could be a NN, tabular, etc. • Critic provides training signal to actor Without actor: use an optimization algorithm Teacher with the critic Can use supervised learning to first approximate the behavior of a different control policy

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC A few examples …

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Dealing with “Long-T erm” Time Dependencies: Resonant Frequency Control in Normal Conducting Cavities RF electron gun at the Fermilab Accelerator Radio frequency quadrupole (RFQ) for the Science and Technology (FAST) facility PIP-II Injector Test Photo: J. Steimel Photo: P. Stabile “long term” in this case means responses lasting many minutes (e.g. 30), with control actions at 0.5 Hz and 1 Hz

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Why does this matter for normal-conducting cavities? The LLRF system will compensate for detuning by increasing forward power

Auralee Edelen, ICFA Mini-Workshop on ML for Particle Accelerators, Feb. 27 – Mar. 3, 2018 at SLAC Why does this matter for normal-conducting cavities? The LLRF system will compensate for detuning by increasing forward power But… • Ability to do this bounded by the amplifier specs • If detuned beyond RF overhead à interrupt normal operations • RF overhead adds to initial machine cost and footprint • Using additional RF power à increasing operational cost • Increased waste heat into cooling system à increasing operational cost