Machine Learning for Orders of Magnitude Speedup in Multi-Objective Optimization of Particle Accelerator Systems.

High-fidelity physics simulations are powerful tools in the design and optimization of charged particle accelerators. However, the computational burden of these simulations often limits their use in practice for design optimization and experiment planning. It also precludes their use as online models tied directly to accelerator operation. We introduce an approach based on machine learning to create nonlinear, fast-executing surrogate models that are informed by a sparse sampling of the physics simulation. The models are 10^6 to 10^7 times more efficient to execute.We also demonstrate that these models can be reliably used with multi-objective optimization to obtain orders-of-magnitude speedup in initial design studies and experiment planning. For example, we required 132 times fewer simulation evaluations to obtain an equivalent solution for our main test case, and initial studies suggest that between 330 to 550 times fewer simulation evaluations are needed when using an iterative retraining process. Our approach enables new ways for high-fidelity particle accelerator simulations to be used, at comparatively little computational cost.

[1]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[2]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[3]  A. L. Edelen Results and Discussion of Recent Applications of Neural Network-Based Approaches to the Modeling and Control of Particle Accelerators , 2018 .

[4]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[5]  Stefano Ermon,et al.  Bayesian Optimization of FEL Performance at LCLS , 2016 .

[6]  Yves Ineichen,et al.  Toward massively parallel multi-objective optimization with application to particle accelerators , 2013 .

[7]  Z.Li,et al.  Advances in Massively Parallel Electromagnetic Simulation Suite ACE3P , 2015 .

[8]  Jean-Luc Vay,et al.  PPPS-2013: Topic 1.2: A domain decomposition method for pseudo-spectral electromagnetic simulations of plasmas , 2013, 2013 Abstracts IEEE International Conference on Plasma Science (ICOPS).

[9]  A. Scheinker Applying Artificial Intelligence to Accelerators , 2018 .

[10]  James P. Sethna,et al.  Online storage ring optimization using dimension-reduction and genetic algorithms , 2018, Physical Review Accelerators and Beams.

[11]  C. K. Allen,et al.  Beam halo definitions based upon moments of the particle distribution , 2002 .

[12]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[13]  J. Shtalenkova,et al.  Online tuning and light source control using a physics-informed Gaussian process Adi , 2019, ArXiv.

[14]  Andreas Adelmann,et al.  On Nonintrusive Uncertainty Quantification and Surrogate Model Construction in Particle Accelerator Modeling , 2019, SIAM/ASA J. Uncertain. Quantification.

[15]  SciDAC advances and applications in computational beam dynamics , 2005 .

[16]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[17]  M. Borland,et al.  Elegant : a flexible SDDS-compliant code for accelerator simulation. , 2000 .

[18]  C. Melton,et al.  Demonstration of Machine Learning-Based Model-Independent Stabilization of Source Properties in Synchrotron Light Sources. , 2019, Physical review letters.

[19]  Marc Parizeau,et al.  DEAP: evolutionary algorithms made easy , 2012, J. Mach. Learn. Res..

[20]  Peter Arbenz,et al.  Parallel general purpose multiobjective optimization framework with application to electron beam dynamics , 2019, Physical Review Accelerators and Beams.

[21]  James V. Candy,et al.  Adaptive and Learning Systems for Signal Processing, Communications, and Control , 2006 .

[22]  J. J. Yang,et al.  Beam dynamics simulation for the high intensity DAEδALUS cyclotrons , 2012, 1209.5864.

[23]  Weifeng Liu,et al.  Adaptive and Learning Systems for Signal Processing, Communication, and Control , 2010 .

[24]  Brendan O'Shea,et al.  Machine learning-based longitudinal phase space prediction of particle accelerators , 2018, Physical Review Accelerators and Beams.

[25]  Xiaoying Pang,et al.  High-Performance Beam Simulator for the LANSCE Linac , 2012 .

[26]  Minghao Song,et al.  Machine learning for design optimization of storage ring nonlinear dynamics , 2019, ArXiv.

[27]  Stefano Ermon,et al.  Accurate Uncertainties for Deep Learning Using Calibrated Regression , 2018, ICML.

[28]  Ralph C. Smith,et al.  Uncertainty Quantification: Theory, Implementation, and Applications , 2013 .

[29]  Wanming Liu,et al.  Research Program and Recent Results at the Argonne Wakefield Accelerator Facility (AWA) , 2017 .

[30]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[31]  D. Ratner,et al.  Machine Learning Models for Optimization and Control of X-ray Free Electron Lasers , 2019 .

[32]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[33]  Yves Roblin,et al.  Innovative Applications of Genetic Algorithms to Problems in Accelerator Physics , 2013 .

[34]  Alexander Scheinker,et al.  Demonstration of Model-Independent Control of the Longitudinal Phase Space of Electron Beams in the Linac-Coherent Light Source with Femtosecond Resolution. , 2018, Physical review letters.

[35]  J. J. Yang,et al.  Proposal for an electron antineutrino disappearance search using high-rate 8Li production and decay. , 2012, Physical review letters.

[36]  S. Ellacott Techniques for the mathematical analysis of neural networks , 1994 .

[37]  N. Eddy,et al.  Initial experimental results of a machine learning-based temperature control system for an RF gun , 2015 .

[38]  Andrei P Shishlo,et al.  Open XAL status report 2016 , 2016 .

[39]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[40]  Ji Qiang,et al.  An Object-Oriented Parallel Particle-in-Cell Code for Beam Dynamics Simulation in Linear Accelerators , 2000, ACM/IEEE SC 1999 Conference (SC'99).

[41]  I. Sobola,et al.  Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates , 2001 .

[42]  Steven M. Lund,et al.  Computational Methods in the Warp Code Framework for Kinetic Simulations of Particle Beams and Plasmas , 2014, IEEE Transactions on Plasma Science.

[43]  S. Lidia,et al.  A Three-Dimensional Quasi-Static Model for High Brightness Beam Dynamics Simulation , 2005 .

[44]  Xiaoying Pang Advances in Proton Linac Online Modeling , 2015 .

[45]  Cosmin Safta,et al.  Uncertainty Quantification Toolkit (UQTk) , 2015 .

[46]  J. E. Stovall,et al.  Modified PARMILA code for new accelerating structures , 1995, Proceedings Particle Accelerator Conference.

[47]  Habib N. Najm,et al.  Numerical Challenges in the Use of Polynomial Chaos Representations for Stochastic Processes , 2005, SIAM J. Sci. Comput..

[48]  R. Ghanem,et al.  Stochastic Finite Elements: A Spectral Approach , 1990 .

[49]  R. French Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[50]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[51]  Andreas Krause,et al.  Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces , 2019, ICML.

[52]  R. Ischebeck,et al.  Opportunities in Machine Learning for Particle Accelerators , 2018, 1811.03172.

[53]  Alexander Scheinker,et al.  Adaptive method for electron bunch profile prediction , 2015 .

[54]  J. Meiss Symplectic maps, variational principles, and transport , 1992 .

[55]  J. Meiss Symplectic maps , 2008 .

[56]  A. L. Edelen S.G. Biedron S.V. Milton J.P. Edelen First Steps Toward Incorporating Image Based Diagnostics Into Particle Accelerator Control Systems Using Convolutional Neural Networks , 2016 .

[57]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[58]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[59]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[60]  R. Barlow,et al.  Medical isotope production with the IsoDAR cyclotron , 2019, Nature Reviews Physics.

[61]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[62]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[63]  MULTI-OBJECTIVE ONLINE OPTIMIZATION OF BEAM LIFETIME AT APS ∗ , 2017 .

[64]  Uldis Locans,et al.  The Dynamic Kernel Scheduler - Part 1 , 2016, Comput. Phys. Commun..

[65]  Matthew F. Dixon,et al.  Bayesian Regression and Gaussian Processes , 2020 .

[66]  Carlos M. Fonseca,et al.  An Improved Dimension-Sweep Algorithm for the Hypervolume Indicator , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[67]  J. Maxson,et al.  Demonstration of Low Emittance in the Cornell Energy Recovery Linac Injector Prototype , 2013, 1304.2708.

[68]  Sandra Biedron,et al.  Using a Neural Network Control Policy for Rapid Switching Between Beam Parameters in an FEL , 2017 .

[69]  K. Halbach,et al.  Superfish-a Computer Program for Evaluation of RF Cavities with Cylindrical Symmetry , 1976 .

[70]  W. Gai,et al.  Short-pulse dielectric two-beam acceleration , 2012, Journal of Plasma Physics.

[71]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[72]  Jean-Luc Vay,et al.  A spectral, quasi-cylindrical and dispersion-free Particle-In-Cell algorithm , 2015, Comput. Phys. Commun..

[73]  G. P. Boicourt,et al.  Revision of and Documentation for the Standard Version of the Poisson Group Codes , 1985, IEEE Transactions on Nuclear Science.

[74]  Stefano Ermon,et al.  Sparse Gaussian Processes for Bayesian Optimization , 2016, UAI.

[75]  Lawrence J. Rybarcyk,et al.  Multi-objective particle swarm and genetic algorithm for the optimization of the LANSCE linac operation , 2014 .

[76]  M. Cho,et al.  Perturbation-minimized triangular bunch for high-transformer ratio using a double dogleg emittance exchange beam line , 2016 .

[77]  D. Edstrom,et al.  Neural Networks for Modeling and Control of Particle Accelerators , 2016, IEEE Transactions on Nuclear Science.

[78]  A. Adelmann On Uncertainty Quantification in Particle Accelerators Modelling , 2015, 1509.08130.

[79]  Nicola Beume,et al.  On the Complexity of Computing the Hypervolume Indicator , 2009, IEEE Transactions on Evolutionary Computation.

[80]  J. Vay,et al.  Noninvariance of space- and time-scale ranges under a Lorentz Transformation and the implications for the study of relativistic interactions. , 2007, Physical review letters.

[81]  P. Arbenz,et al.  A Parallel General Purpose Multi-Objective Optimization Framework, with Application to Beam Dynamics , 2013, 1302.2889.

[82]  Charles Sinclair,et al.  Multivariate optimization of a high brightness dc gun photoinjector , 2005 .

[83]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[84]  Christopher. Simons,et al.  Machine learning with Python , 2017 .