Gradient Population Optimization: A Tensorflow-Based Heterogeneous Non-Von-Neumann Paradigm for Large-Scale Search

This paper presents a novel scalable algorithm, Gradient Population Optimization (GPO), which is specifically designed to optimize cost functions with extremely high dimensionality. GPO uses the Tensorflow platform, a non-von-Neumann computation model, which implements dataflow graphs on heterogeneous computing hardware (e.g., multi-core central processing unit, graphics processing unit (GPU), and field-programmable gate array) in order to perform massively parallel processing tasks on scalable platforms, such as the cloud. GPO is based on the combination of population-based dynamics with gradient-based determinism, in which a coupling term is introduced between the local and global corrections to the positions of population’s agents’ positions. The GPO exhibited excellent performance in most of the standard benchmark functions that were tested. In particular, GPO demonstrated superb scalability in solving large-scale optimization problems using GPU-hardware-accelerated computing platform, positing the algorithm as an effective strategy for real-life massive scale problems, such as machine learning, data mining, and modeling wireless communication systems, such as 5G and massive MIMO.

[1]  Gianluca Francini,et al.  Optimized Deep Neural Networks for Real-Time Object Classification on Embedded GPUs , 2017 .

[2]  Jack B. Dennis,et al.  Data Flow Supercomputers , 1980, Computer.

[3]  Koffka Khan,et al.  A Comparison of BA, GA, PSO, BP and LM for Training Feed forward Neural Networks in e-Learning Context , 2012 .

[4]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[5]  He Jiang.  An hybrid heuristic algorithm for the Two-Echelon Vehicle Routing Problem , 2012 .

[6]  Alex A. Freitas,et al.  A hybrid PSO/ACO algorithm for discovering classification rules in data mining , 2008 .

[7]  Peter Goldsborough,et al.  A Tour of TensorFlow , 2016, ArXiv.

[8]  Jianming Deng,et al.  Analysis of Population Diversity of Dynamic Probabilistic Particle Swarm Optimization Algorithms , 2014 .

[9]  Adil M. Bagirov,et al.  Local Optimization Method with Global Multidimensional Search , 2005, J. Glob. Optim..

[10]  Hüseyin Gürüler,et al.  Rapid Automated Classification of Anesthetic Depth Levels using GPU Based Parallelization of Neural Networks , 2015, Journal of medical systems.

[11]  R. Eberhart,et al.  Empirical study of particle swarm optimization , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[12]  Shin-Dug Kim,et al.  Accelerating the pre-processing stages of JPEG encoder on a heterogenous system using OpenCL , 2015, 2015 IEEE/ACIS 16th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD).

[13]  Mohamed Zahran,et al.  Heterogeneous computing , 2017, Commun. ACM.

[14]  V. Kober,et al.  Analysis of the gradient descent method in problems of the signals and images restoration , 2015, Pattern Recognition and Image Analysis.

[15]  Jing J. Liang,et al.  Comprehensive learning particle swarm optimizer for global optimization of multimodal functions , 2006, IEEE Transactions on Evolutionary Computation.

[16]  Yahia M. M. Antar,et al.  A novel massively-parallel processing framework for real-time MIMO and smart antenna array beam control , 2017, 2017 Computing and Electromagnetics International Workshop (CEM).

[17]  Tao Xiang,et al.  Real-Time Incompressible Fluid Simulation on the GPU , 2015, Int. J. Comput. Games Technol..

[18]  Mauro Birattari,et al.  Swarm Intelligence , 2012, Lecture Notes in Computer Science.

[19]  Henri E. Bal,et al.  Cashmere: Heterogeneous Many-Core Computing , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium.

[20]  William W. Wadge,et al.  Lucid, the dataflow programming language , 1985 .

[21]  S. Shamsuddin,et al.  Fusion Global-Local-Topology Particle Swarm Optimization for Global Optimization Problems , 2014 .

[22]  Martin Pelikan,et al.  Hierarchical Bayesian optimization algorithm: toward a new generation of evolutionary algorithms , 2010, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[23]  Y. Rahmat-Samii,et al.  Particle swarm, genetic algorithm, and their hybrids: optimization of a profiled corrugated horn antenna , 2002, IEEE Antennas and Propagation Society International Symposium (IEEE Cat. No.02CH37313).

[24]  Vadlamani Ravi,et al.  A particle swarm optimization-threshold accepting hybrid algorithm for unconstrained optimization , 2013 .

[25]  Derviş Karaboğa,et al.  NEURAL NETWORKS TRAINING BY ARTIFICIAL BEE COLONY ALGORITHM ON PATTERN CLASSIFICATION , 2009 .

[26]  Wayne Luk,et al.  A comparison of CPUs, GPUs, FPGAs, and massively parallel processor arrays for random number generation , 2009, FPGA '09.

[27]  Shengxiang Yang,et al.  A particle swarm optimization based memetic algorithm for dynamic optimization problems , 2010, Natural Computing.

[28]  Yudong Zhang,et al.  A Comprehensive Survey on Particle Swarm Optimization Algorithm and Its Applications , 2015 .

[29]  Carmelo J. A. Bastos Filho,et al.  Clan Particle Swarm Optimization , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[30]  Theo Ungerer,et al.  Processor architecture - from dataflow to superscalar and beyond , 1999 .

[31]  Scott B. Baden,et al.  Scalable Heterogeneous CPU-GPU Computations for Unstructured Tetrahedral Meshes , 2015, IEEE Micro.

[32]  Doug DeGroot,et al.  Parallel Machines: Parallel Machine Languages: The Emergence of Hybrid Dataflow Computer Architectures , 1990 .

[33]  Medhat A. Tawfeek,et al.  Hybrid Algorithm Based on Swarm Intelligence Techniques for Dynamic Tasks Scheduling in Cloud Computing , 2016 .

[34]  J. Kennedy,et al.  Population structure and particle swarm performance , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[35]  Ying Tan,et al.  GPU-based Parallel Implementation of Swarm Intelligence Algorithms , 2016 .

[36]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[37]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[38]  Ravi Iyer,et al.  Heterogeneous Computing [Guest editors' introduction] , 2015, IEEE Micro.

[39]  John Paul T. Yusiong,et al.  Optimizing Artificial Neural Networks using Cat Swarm Optimization Algorithm , 2012 .

[40]  Kok Lay Teo,et al.  A Hybrid Descent Method for Global Optimization , 2004, J. Glob. Optim..

[41]  Yongling Zheng,et al.  On the convergence analysis and parameter selection in particle swarm optimization , 2003, Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.03EX693).

[42]  Russell C. Eberhart,et al.  Tracking and optimizing dynamic systems with particle swarms , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[43]  Alasdair N. Houston,et al.  Visual Simulation of Soil-Microbial System Using GPGPU Technology , 2015, Comput..