Multidirectional search: a direct search algorithm for parallel machines

Multi-Directional Search: A Direct Search Algorithm for Parallel Machines by Virginia Joanne Torczon In recent years there has been a great deal of interest in the development of optimization algorithms which exploit the computational power of parallel computer architectures. We have developed a new direct search algorithm, which we call multidirectional search, that is ideally suited for parallel computation. Our algorithm belongs to the class of direct search methods, a class of optimization algorithms which neither compute nor approximate any derivatives of the objective function. Our work, in fact, was inspired by the simplex method of Spendley, Hext, and Himsworth, and the simplex method of Nelder and Mead. The multi-directional search algorithm is inherently parallel. The basic idea of the algorithm is to perform concurrent searches in multiple directions. These searches are free of any interdependencies, so the information required can be computed in parallel. A central result of our work is the convergence analysis for our algorithm. By requiring only that the function be continuously di erentiable over a bounded level set, we can prove that a subsequence of the points generated by the multi-directional search algorithm converges to a stationary point of the objective function. This is of great interest since we know of few convergence results for practical direct search algorithms. We also present numerical results indicating that the multi-directional search algorithm is robust, even in the presence of noise. Our results include comparisons with the Nelder-Mead simplex algorithm, the method of steepest descent, and a quasi-Newton method. One surprising conclusion of our numerical tests is that the Nelder-Mead simplex algorithm is not robust. We close with some comments about future directions of research. Acknowledgments First, I would like to thank my family. Without their unwavering support I would never have made it this far. If they ever doubted that a historian could also be a mathematician, they never let it show. I would like to thank the members of my committee; each one has made an important contribution to this work. Andy Boyd convinced rst me, and then my chairman, that the last argument needed to complete the convergence proof was correct; he then pushed me to a most satisfying generalization. Ken Kennedy shared his knowledge of parallel computation and pointed me towards a very clever load balancing scheme. Richard Tapia deserves special credit for ever agreeing to accept me into the graduate program. He is also responsible for introducing me to optimization theory. Of late, he has shared his knowledge of the theory for descent methods, which has helped shaped the nal form of my arguments. I have also enjoyed our many conversations on issues as diverse as music, minorities, and machismo. I owe special thanks to my chairman, John Dennis. He asked me to stay, gave me this problem to work on, and made me tough. I owe a great deal to his inimitable style of motivation: His unerring response to my latest result would be either \You're wrong." or \That can't possibly be true." | which of course made me all the more determined to prove that I was, in fact, right. Now he moans that I am always right, as if there were no connection. I would also like to thank the many other people at Rice from whom I have learned much: Richard Byrd helped isolate the last link needed to complete my convergence result. Linda Torczon and Keith Cooper have taught me much about computers and computation. Nancy Ginsburg shared the early, di cult years of graduate school. v Cathy Samuelsen took up where Nancy left o . Cathy and Karen Williamson su ered through numerous drafts of my thesis; it is better for their e orts. Finally, I would like to thank my husband, Michael Lewis. He was the rst, true believer in this work; I managed to convince him that I was right before I had even convinced myself. His faith in my mathematical abilities has pushed me further than anyone else thought possible. His good taste has left its mark on all this work.