Multi-agent system and reinforcement learning approach for distributed intelligence in a flexible smart manufacturing system

Abstract Personalized production has emerged as a result of the increasing customer demand for more personalized products. Personalized production systems carry a greater amount of uncertainty and variability when compared with traditional manufacturing systems. In this paper, we present a smart manufacturing system using a multi-agent system and reinforcement learning, which is characterized by machines with intelligent agents to enable a system to have autonomy of decision making, sociability to interact with other systems, and intelligence to learn dynamically changing environments. In the proposed system, machines with intelligent agents evaluate the priorities of jobs and distribute them through negotiation. In addition, we propose methods for machines with intelligent agents to learn to make better decisions. The performance of the proposed system and the dispatching rule is demonstrated by comparing the results of the scheduling problem with early completion, productivity, and delay. The obtained results show that the manufacturing system with distributed artificial intelligence is competitive in a dynamic environment.

[1]  Tapas K. Das,et al.  A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem , 2005, Simul. Model. Pract. Theory.

[2]  Ali Vatankhah Barenji,et al.  A dynamic multi-agent-based scheduling approach for SMEs , 2017 .

[3]  Paulo Leitão,et al.  Past, Present, and Future of Industrial Agent Applications , 2013, IEEE Transactions on Industrial Informatics.

[4]  Andrew Kusiak,et al.  Data-driven smart manufacturing , 2018, Journal of Manufacturing Systems.

[5]  Wilfried Brauer,et al.  Multi-machine scheduling-a multi-agent learning approach , 1998, Proceedings International Conference on Multi Agent Systems (Cat. No.98EX160).

[6]  Aydin Nassehi,et al.  Anarchic manufacturing: Distributed control for product transition , 2020, Journal of Manufacturing Systems.

[7]  Richard Y. K. Fung,et al.  Integrated process planning and scheduling by an agent-based ant colony optimization , 2010, Comput. Ind. Eng..

[8]  Ari P. J. Vepsalainen.  Priority rules for job shops with weighted tardiness costs , 1987 .

[9]  Paulo Leitão,et al.  Key Contributing Factors to the Acceptance of Agents in Industrial Environments , 2017, IEEE Transactions on Industrial Informatics.

[10]  Charles E. Dickerson,et al.  Dynamic Production System Identification for Smart Manufacturing Systems. , 2018, Journal of manufacturing systems.

[11]  Aydin Nassehi,et al.  Balancing multiple objectives with anarchic manufacturing , 2019 .

[12]  Yi-Chi Wang,et al.  Application of reinforcement learning for agent-based production scheduling , 2005, Eng. Appl. Artif. Intell..

[13]  Jamal Shahrabi,et al.  A reinforcement learning approach to parameter estimation in dynamic job shop scheduling , 2017, Comput. Ind. Eng..

[14]  Weiming Shen,et al.  MetaMorph: An adaptive agent-based architecture for intelligent manufacturing , 1999 .

[15]  Chun Jin,et al.  Deep reinforcement learning for a color-batching resequencing problem , 2020 .

[16]  Lida Xu,et al.  Internet of Things for Enterprise Systems of Modern Manufacturing , 2014, IEEE Transactions on Industrial Informatics.

[17]  Lamjed Ben Said,et al.  On-line self-adaptive framework for tailoring a neural-agent learning model addressing dynamic real-time scheduling problems , 2017 .

[18]  Robert W. Brennan,et al.  An architecture for metamorphic control of holonic manufacturing systems , 2001, Comput. Ind..

[19]  Luc Bongaerts,et al.  Designing Holonic manufacturing systems , 1998 .

[20]  Henri Pierreval,et al.  Real time selection of scheduling rules and knowledge extraction via dynamically controlled data mining , 2010 .

[21]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[22]  Kenneth R. Baker,et al.  Sequencing Rules and Due-Date Assignments in a Job Shop , 1984 .

[23]  Luc Bongaerts,et al.  Reference architecture for holonic manufacturing systems: PROSA , 1998 .

[24]  Chris N. Potts,et al.  A Branch and Bound Algorithm for the Total Weighted Tardiness Problem , 1985, Oper. Res..

[25]  Sicheng Zhang,et al.  Integrated process planning and scheduling – multi-agent system with two-stage ant colony optimisation algorithm , 2012 .

[26]  Daqiang Zhang,et al.  Towards smart factory for industry 4.0: a self-organized multi-agent system with big data based feedback and coordination , 2016, Comput. Networks.

[27]  S. S. Panwalkar,et al.  A Survey of Scheduling Rules , 1977, Oper. Res..

[28]  Zhenyu Liu,et al.  Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network , 2020 .

[29]  Qing Chang,et al.  Simulation study on reward function of reinforcement learning in gantry work cell scheduling , 2019, Journal of Manufacturing Systems.

[30]  Dimitris Mourtzis,et al.  Challenges and future perspectives for the life cycle of manufacturing networks in the mass customisation era , 2016, Logist. Res..

[31]  Ching-Jen Huang,et al.  A multi-agent-based negotiation approach for parallel machine scheduling with multi-objectives in an electro-etching process , 2012 .

[32]  Ahmed Kouider,et al.  Distributed multi-agent scheduling and control system for robotic flexible assembly cells , 2019, J. Intell. Manuf..

[33]  S. Zhang,et al.  Integrated process planning and scheduling: an enhanced ant colony optimization heuristic with parameter tuning , 2018, J. Intell. Manuf..

[34]  WanJiafu,et al.  Towards smart factory for industry 4.0 , 2016 .

[35]  Fred Glover,et al.  Tabu search methods for a single machine scheduling problem , 1991, J. Intell. Manuf..

[36]  Lihui Wang.  Integrated design-to-control approach for holonic manufacturing systems , 2001 .

[37]  Michael Pinedo,et al.  Scheduling jobs on parallel machines with sequence-dependent setup times , 1997, Eur. J. Oper. Res..

[38]  Sicheng Zhang,et al.  Flexible job-shop scheduling/rescheduling in dynamic environment: a hybrid MAS/ACO approach , 2017, Int. J. Prod. Res..

[39]  David B. Shmoys,et al.  Scheduling to Minimize Average Completion Time: Off-Line and On-Line Approximation Algorithms , 1997, Math. Oper. Res..

[40]  B. Pascal,et al.  A holonic approach for manufacturing execution system design: An industrial application , 2007, 2007 IEEE Conference on Emerging Technologies and Factory Automation (EFTA 2007).

[41]  José Barbosa,et al.  Dynamic self-organization in holonic multi-agent manufacturing systems: The ADACOR evolution , 2015, Comput. Ind..

[42]  Duck Young Kim,et al.  An Extended Agent Communication Framework for Rapid Reconfiguration of Distributed Manufacturing Systems , 2019, IEEE Transactions on Industrial Informatics.

[43]  Aydin Nassehi,et al.  Embracing complicatedness and complexity with Anarchic Manufacturing , 2019 .

[44]  Nelson Rodrigues,et al.  Multiagent System Integrating Process and Quality Control in a Factory Producing Laundry Washing Machines , 2015, IEEE Transactions on Industrial Informatics.

[45]  Weiming Shen,et al.  Learning in Agent-Based Manufacturing Systems , 1998 .

[46]  Mozafar Saadat,et al.  A New Agents-Based Model for Dynamic Job Allocation in Manufacturing Shopfloors , 2012, IEEE Systems Journal.

[47]  Li Li,et al.  ACO-based multi-objective scheduling of parallel batch processing machines with advanced process control constraints , 2009 .

[48]  Chao-Ton Su,et al.  Real-time scheduling for a smart factory using a reinforcement learning approach , 2018, Comput. Ind. Eng..

[49]  Michael Pinedo,et al.  A heuristic to minimize the total weighted tardiness with sequence-dependent setups , 1997 .