Simulation based optimization of a stochastic supply chain considering supplier disruption: An agent-based modeling and reinforcement learning

Many researchers and practitioners in the recent years have been attracted to investigate the role of uncertainties in the supply chain management concept. In this paper a multi-period stochastic supply chain with demand uncertainty and supplier disruption is modeled. In the model, two types of retailers including risk sensitive and risk neutral, with many capacitated suppliers are considered. Autonomous retailers have three choices to satisfy demands: ordering from primary suppliers, reserved suppliers and spot market. The goal is to find the best behavior of the risk sensitive retailer, regarding the forward and option contracts, during several contract periods based on the profit function. Hence, an agent-based simulation approach has been developed to simulate the supply chain and transactions between retailers and unreliable suppliers. In addition, a Q-learning approach (as a method of reinforcement learning) has been developed to optimize the simulation procedure. Furthermore, different configurations for simulation procedure are analyzed. The R-netlogo package is used to implement the algorithm. Also a numerical example has been solved using the proposed simulation-optimization approach. Several sensitivity analyzes are conducted regarding different parameters of the model. Comparison of the numerical results with a genetic algorithm shows a significant efficiency of the proposed Q-leaning approach.

[1]  Marianthi G. Ierapetritou,et al.  Hybrid simulation based optimization approach for supply chain management , 2012, Comput. Chem. Eng..

[2]  Ohbyung Kwon,et al.  Corrigendum to "MACE-SCM: A multi-agent and case-based reasoning collaboration mechanisms for supply chain management under supply and demand uncertainties" [Expert Systems with Applications 33 (3) (2007) 690-705] , 2008, Expert Syst. Appl..

[3]  Ibrahim Dogan,et al.  A reinforcement learning approach to competitive ordering and pricing problem , 2015, Expert Syst. J. Knowl. Eng..

[4]  Yasemin Merzifonluoglu,et al.  Production , Manufacturing and Logistics Integrated demand and procurement portfolio management with spot market volatility and option contracts , 2016 .

[5]  Inneke Van Nieuwenhuyse,et al.  Simulation optimization in inventory replenishment: a classification , 2015 .

[6]  Haifeng Wang,et al.  Optimal Inventory Decisions in a Multi-period News Vendor Problem with Partially Observed Markovian Supply Capacities , 2008, Eur. J. Oper. Res..

[7]  Abhijit Gosavi,et al.  Reinforcement learning for long-run average cost , 2004, Eur. J. Oper. Res..

[8]  Charles M. Macal,et al.  Everything you need to know about agent-based modelling and simulation , 2016, J. Simulation.

[9]  Zhaohan Sheng,et al.  Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system , 2009, Expert Syst. Appl..

[10]  Mamata Jenamani,et al.  Sourcing decision under disruption risk with supply and demand uncertainty: A newsvendor approach , 2016, Ann. Oper. Res..

[11]  J. Winch,et al.  Supply Chain Management: Strategy, Planning, and Operation , 2003 .

[12]  Seyed Hessameddin Zegordi,et al.  A reinforcement learning model for supply chain ordering management: An application to the beer game , 2008, Decis. Support Syst..

[13]  Parham Azimi,et al.  Designing of an intelligent self-adaptive model for supply chain ordering management system , 2015, Eng. Appl. Artif. Intell..

[14]  Yasemin Merzifonluoglu,et al.  Risk averse supply portfolio selection with supply, demand and spot market volatility , 2015 .

[15]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[16]  Azad M. Madni,et al.  Integrated Agent-based modeling and optimization in complex systems analysis , 2014, CSER.

[17]  Yasemin Merzifonluoglu,et al.  Impact of risk aversion and backup supplier on sourcing decisions of a firm , 2015 .

[18]  Yasemin Merzifonluoglu,et al.  Newsvendor problem with multiple unreliable suppliers , 2014 .

[19]  Jan C. Thiele.  R Marries NetLogo: Introduction to the RNetLogo Package , 2014 .

[20]  Markus Rabe,et al.  A Reinforcement Learning approach for a Decision Support System for logistics networks , 2015, 2015 Winter Simulation Conference (WSC).

[21]  Chang Ouk Kim,et al.  Multi-agent based distributed inventory control model , 2010, Expert Syst. Appl..

[22]  Martin Purvis,et al.  A Combined Modelling Approach for Multi-Agent Collaborative Planning in Global Supply Chains , 2015, 2015 8th International Symposium on Computational Intelligence and Design (ISCID).

[23]  Ferdinando Chiacchio,et al.  Agent-Based Modeling of the Immune System: NetLogo, a Promising Framework , 2014, BioMed research international.

[24]  Mamata Jenamani,et al.  Sourcing under supply disruption with capacity‐constrained suppliers , 2013 .

[25]  Ran Liu,et al.  Simulation-based optimisation approach for the stochastic two-echelon logistics problem , 2017 .

[26]  Gitae Kim,et al.  Optimal inventory control in a multi-period newsvendor problem with non-stationary demand , 2015, Adv. Eng. Informatics.

[27]  Hasan Selim,et al.  A Multi-objective, simulation-based optimization framework for supply chains with premium freights , 2017, Expert Syst. Appl..

[28]  Matthew J. Sobel,et al.  Inventory Control with an Exponential Utility Criterion , 1992, Oper. Res..

[29]  Gang Zhao,et al.  Analyses about efficiency of reinforcement learning to supply chain ordering management , 2012, IEEE 10th International Conference on Industrial Informatics.

[30]  M. Densing,et al.  Dispatch planning using newsvendor dual problems and occupation times: Application to hydropower , 2013, Eur. J. Oper. Res..

[31]  Kun Chang Lee,et al.  MACE-SCM: A multi-agent and case-based reasoning collaboration mechanism for supply chain management under supply and demand uncertainties , 2007, Expert Syst. Appl..