Neural networks in computational materials science: training algorithms

Neural networks can be used in principle in an unbiased way for a multitude of pattern recognition and interpolation problems within computational material science. Reliably finding the weights of large feed-forward neural networks with both accuracy and speed is crucial to their use. In this paper, the rate of convergence of numerous optimization techniques that can be used to determine the weights is compared for two problems related to the construction of atomistic potentials. Techniques considered were back propagation (steepest descent), conjugate gradient methods, real-string genetic algorithms, simulated annealing and a new swarm search algorithm. For small networks, where only a few optimal solutions exist, we find that conjugate-gradient methods are most successful. However, for larger networks where the parameter space to be searched is more complex, a hybrid scheme is most effective; genetic algorithm or simulated annealing to find a good initial starting set of weights, followed by a conjugate-gradient approach to home in on a final solution. These hybrid approaches are now our method of choice for training large networks.