Hyper-heuristics using multi-armed bandit models for multi-objective optimization

Abstract In this work, we explore different multi-armed bandit-based hyper-heuristics applied to the multi-objective permutation flow shop problem. It is a scheduling problem which has been extensively studied due to its relevance for industrial engineering. Three multi-armed bandit basic formulations are used in the hyper-heuristic selection mechanism: (i) classic, (ii) restless, and (iii) contextual. All the three approaches are considered online selection perturbative hyper-heuristics as they are designed to choose the low level heuristic (crossover and mutation operators) that should be applied to modify each solution during the search process. Performances of the proposed approaches combined with MOEA/DD (Multi-objective Evolutionary Algorithm based on Dominance and Decomposition) are evaluated using the hypervolume indicator and nonparametric statistical tests on a set of 220 problem instances with two and three objectives. The best proposed approach (contextual with a linear realizability assumption) is compared with a stand-alone version of MOEA/DD using the best considered crossover and mutation operator. It is also compared with three state-of-the-art multi-objective algorithms. Results show that this best approach is able to outperform MOEA/DD in scenarios with two and three objectives and by encompassing both, Pareto and decomposition strategies, is competitive with the state of the art in those scenarios.

[1]  Pablo Moscato,et al.  Memetic Algorithms for Business Analytics and Data Science: A Brief Survey , 2019, Business and Consumer Analytics: New Ideas.

[2]  Donald A. Berry,et al.  Bandit Problems: Sequential Allocation of Experiments. , 1986 .

[3]  Eckart Zitzler,et al.  Indicator-Based Selection in Multiobjective Search , 2004, PPSN.

[4]  Patrick De Causmaecker,et al.  An investigation on the generality level of selection hyper-heuristics under different empirical conditions , 2013, Appl. Soft Comput..

[5]  W. J. Conover,et al.  Practical Nonparametric Statistics , 1972 .

[6]  Rong-Qiang Zeng,et al.  Solving bi-objective flow shop problem with hybrid path relinking algorithm , 2013, Appl. Soft Comput..

[7]  Edmund K. Burke,et al.  Recent advances in selection hyper-heuristics , 2020, Eur. J. Oper. Res..

[8]  D. Gong,et al.  Solving the blocking flow shop scheduling problem with makespan using a modified fruit fly optimisation algorithm , 2016 .

[9]  Lihong Li,et al.  An Empirical Evaluation of Thompson Sampling , 2011, NIPS.

[10]  S. N. Sivanandam,et al.  Introduction to genetic algorithms , 2007 .

[11]  Qingfu Zhang,et al.  Adaptive Operator Selection With Bandits for a Multiobjective Evolutionary Algorithm Based on Decomposition , 2014, IEEE Transactions on Evolutionary Computation.

[12]  Li Zhou,et al.  A Survey on Contextual Multi-armed Bandits , 2015, ArXiv.

[13]  Wali Khan Mashwani,et al.  A decomposition-based hybrid multiobjective evolutionary algorithm with dynamic resource allocation , 2012, Appl. Soft Comput..

[14]  John Langford,et al.  Contextual Bandit Algorithms with Supervised Learning Guarantees , 2010, AISTATS.

[15]  Sébastien Bubeck,et al.  Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems , 2012, Found. Trends Mach. Learn..

[16]  Qingfu Zhang,et al.  Efficient Nondomination Level Update Method for Steady-State Evolutionary Multiobjective Optimization , 2017, IEEE Transactions on Cybernetics.

[17]  Joseph Y.-T. Leung,et al.  Minimizing Total Tardiness on One Machine is NP-Hard , 1990, Math. Oper. Res..

[18]  Leslie Pérez Cáceres,et al.  The irace package: Iterated racing for automatic algorithm configuration , 2016 .

[19]  Amir H. Gandomi,et al.  Evolutionary Many-Objective Algorithms for Combinatorial Optimization Problems: A Comparative Study , 2020, Archives of Computational Methods in Engineering.

[20]  Rubén Ruiz,et al.  A Review and Evaluation of Multiobjective Algorithms for the Flowshop Scheduling Problem , 2008, INFORMS J. Comput..

[21]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[22]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[23]  Michel Gendreau,et al.  Hyper-heuristics: a survey of the state of the art , 2013, J. Oper. Res. Soc..

[24]  George Angelos Papadopoulos,et al.  Generic hybridization of MOEA/D with learning for permutation flow shop scheduling problem , 2017, 2017 8th International Conference on Information, Intelligence, Systems & Applications (IISA).

[25]  Gabriela Ochoa,et al.  On the automatic discovery of variants of the NEH procedure for flow shop scheduling using genetic programming , 2011, J. Oper. Res. Soc..

[26]  Dipti Srinivasan,et al.  A Survey of Multiobjective Evolutionary Algorithms Based on Decomposition , 2017, IEEE Transactions on Evolutionary Computation.

[27]  James Smith,et al.  A tutorial for competent memetic algorithms: model, taxonomy, and design issues , 2005, IEEE Transactions on Evolutionary Computation.

[28]  Quan-Ke Pan,et al.  Evolutionary multi-objective blocking lot-streaming flow shop scheduling with interval processing time , 2016, Appl. Soft Comput..

[29]  Thomas Stützle,et al.  An Iterated Greedy heuristic for the sequence dependent setup times flowshop problem with makespan and weighted tardiness objectives , 2008, Eur. J. Oper. Res..

[30]  Kenneth R. Baker,et al.  Principles of Sequencing and Scheduling , 2018 .

[31]  Rubén Ruiz,et al.  Restarted Iterated Pareto Greedy algorithm for multi-objective flowshop scheduling problems , 2011, Comput. Oper. Res..

[32]  Chongguo Li,et al.  Diversity Metrics in Multi-objective Optimization: Review and Perspective , 2007, 2007 IEEE International Conference on Integration Technology.

[33]  John E. Dennis,et al.  Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems , 1998, SIAM J. Optim..

[34]  Xiao-Yan Sun,et al.  A discrete artificial bee colony algorithm incorporating differential evolution for the flow-shop scheduling problem with blocking , 2015 .

[35]  Cecilia E. Nugraheni,et al.  A Tabu-Search Based Constructive Hyper-heuristics for Scheduling Problems in Textile Industry , 2017 .

[36]  Ruhul A. Sarker,et al.  A genetic algorithm for permutation flow shop scheduling under make to stock production system , 2015, Comput. Ind. Eng..

[37]  Osho Ilamah,et al.  A multiobjective dominance and decomposition algorithm for reservoir model history matching , 2019 .

[38]  Yaochu Jin,et al.  Evolutionary Multiobjective Blocking Lot-Streaming Flow Shop Scheduling With Machine Breakdowns , 2019, IEEE Transactions on Cybernetics.

[39]  Mehmet Mutlu Yenisey,et al.  Multi-objective permutation flow shop scheduling problem: Literature review, classification and current trends , 2014 .

[40]  Qingfu Zhang,et al.  An Evolutionary Many-Objective Optimization Algorithm Based on Dominance and Decomposition , 2015, IEEE Transactions on Evolutionary Computation.

[41]  Xin-She Yang,et al.  A Discrete Firefly Algorithm for the Multi-Objective Hybrid Flowshop Scheduling Problems , 2014, IEEE Transactions on Evolutionary Computation.

[42]  Thomas Stützle,et al.  Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization , 2010, Experimental Methods for the Analysis of Optimization Algorithms.

[43]  Xiangtao Li,et al.  Multiobjective Local Search Algorithm-Based Decomposition for Multiobjective Permutation Flow Shop Scheduling Problem , 2015, IEEE Transactions on Engineering Management.

[44]  Aurora Trinidad Ramirez Pozo,et al.  A New Hyper-Heuristic Based on a Restless Multi-armed Bandit for Multi-objective Optimization , 2017, 2017 Brazilian Conference on Intelligent Systems (BRACIS).

[45]  Carolina P. de Almeida,et al.  Multi-armed Bandit Based Hyper-Heuristics for the Permutation Flow Shop Problem , 2018, 2018 7th Brazilian Conference on Intelligent Systems (BRACIS).

[46]  H. Ishibuchi,et al.  Multi-objective genetic algorithm and its applications to flowshop scheduling , 1996 .

[47]  J E C Arroyo,et al.  A partial enumeration heuristic for multi-objective flowshop scheduling problems , 2004, J. Oper. Res. Soc..

[48]  Peter Auer,et al.  Using Confidence Bounds for Exploitation-Exploration Trade-offs , 2003, J. Mach. Learn. Res..

[49]  Qing Zhao,et al.  Learning and sharing in a changing world: Non-Bayesian restless bandit with multiple players , 2011, 2011 Information Theory and Applications Workshop.

[50]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[51]  Kevin Kok Wai Wong,et al.  Classification of adaptive memetic algorithms: a comparative study , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[52]  Demosthenis Teneketzis,et al.  Multi-Armed Bandit Problems , 2008 .

[53]  Bernard Manderick,et al.  Bernoulli bandits: an empirical comparison , 2015, ESANN.

[54]  José Elias Claudio Arroyo,et al.  A GRASP heuristic for the multi-objective permutation flowshop scheduling problem , 2011 .

[55]  Éric D. Taillard,et al.  Benchmarks for basic scheduling problems , 1993 .

[56]  Matthias Ehrgott,et al.  A discussion of scalarization techniques for multiple objective integer programming , 2006, Ann. Oper. Res..

[57]  Ravi Sethi,et al.  The Complexity of Flowshop and Jobshop Scheduling , 1976, Math. Oper. Res..

[58]  F. Liu,et al.  An improved multi-objective memetic algorithm for bi-objective permutation flow shop scheduling , 2017, 2017 International Conference on Service Systems and Service Management.

[59]  Thomas Stützle,et al.  A simple and effective iterated greedy algorithm for the permutation flowshop scheduling problem , 2007, Eur. J. Oper. Res..

[60]  Marco Laumanns,et al.  SPEA2: Improving the Strength Pareto Evolutionary Algorithm For Multiobjective Optimization , 2002 .

[61]  Jianyong Sun,et al.  A novel hybrid multi-objective artificial bee colony algorithm for blocking lot-streaming flow shop scheduling problems , 2018, Knowl. Based Syst..

[62]  Thomas Stützle,et al.  Combining Two Search Paradigms for Multi-objective Optimization: Two-Phase and Pareto Local Search , 2013, Hybrid Metaheuristics.

[63]  Graham Kendall,et al.  Choice function based hyper-heuristics for multi-objective optimization , 2015, Appl. Soft Comput..

[64]  Hisao Ishibuchi,et al.  A multi-objective genetic local search algorithm and its application to flowshop scheduling , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[65]  Alexander Mendiburu,et al.  A review of distances for the Mallows and Generalized Mallows estimation of distribution algorithms , 2015, Comput. Optim. Appl..

[66]  André Gustavo dos Santos,et al.  An Iterated Greedy Heuristic for Simultaneous Lot-Sizing and Scheduling Problem in Production Flow Shop Environments , 2014, EvoCOP.