A CUSTOMIZABLE REINFORCEMENT LEARNING ENVIRONMENT FOR SEMICONDUCTOR FAB SIMULATION

Reinforcement learning based methods are increasingly used to solve NP-hard combinatorial optimization problems. By learning from the problem structure, or the characteristics of instances, the approach has high potential compared to alternative techniques solving all instances from scratch. This work introduces a novel framework for creating (deep) reinforcement learning environments simulating up to real-world scale semiconductor fab scheduling problem instances. The highly configurable framework supports creating single- and multi-agent environments where the simulation factory is either partially or fully controlled by the learning agents. The action and observation spaces and the reward function are customizable based on pre-defined features. Our toolkit creates environments with a standard interface that can be integrated with various algorithms in a few minutes. The simulated datasets may involve challenging features like downtimes, batching, rework, and sequence-dependent setups. These can also be turned off and simulated datasets be automatically downscaled during the prototyping phase.

[1]  Mohammed M. S. El-Kholany,et al.  A Customizable Simulator for Artificial Intelligence Research to Schedule Semiconductor Fabs , 2022, 2022 33rd Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC).

[2]  M. Tóth,et al.  The semiconductor shortage and its implication for euro area trade, production and prices , 2021 .

[3]  M. Gebser,et al.  A Reinforcement Learning Environment For Job-Shop Scheduling , 2021, ArXiv.

[4]  Wenxia Guo,et al.  Cloud Resource Scheduling With Deep Reinforcement Learning and Imitation Learning , 2021, IEEE Internet of Things Journal.

[5]  Adar Kalir,et al.  SMT2020—A Semiconductor Manufacturing Testbed , 2020, IEEE Transactions on Semiconductor Manufacturing.

[6]  Doug Suerich,et al.  Reinforcement Learning for Efficient Scheduling in Complex Semiconductor Equipment , 2020, 2020 31st Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC).

[7]  Gisela Lanza,et al.  Designing an adaptive production control system using reinforcement learning , 2020, Journal of Intelligent Manufacturing.

[8]  Jaeseok Huh,et al.  A Reinforcement Learning Approach to Robust Scheduling of Semiconductor Manufacturing Facilities , 2020, IEEE Transactions on Automation Science and Engineering.

[9]  Quoc V. Le,et al.  Chip Placement with Deep Reinforcement Learning , 2020, ArXiv.

[10]  T. Başar,et al.  Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms , 2019, Handbook of Reinforcement Learning and Control.

[11]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[12]  J. Schulman,et al.  OpenAI Gym , 2016, ArXiv.

[13]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[14]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[15]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[16]  Éric D. Taillard,et al.  Benchmarks for basic scheduling problems , 1993 .

[17]  Gülcin Ermis,et al.  Reinforcement Learning for Route Optimization with Robustness Guarantees , 2021, IJCAI.

[18]  A. Gleave,et al.  Stable-Baselines3: Reliable Reinforcement Learning Implementations , 2021, J. Mach. Learn. Res..

[19]  Bohyung Paeng,et al.  Deep Reinforcement Learning for Minimizing Tardiness in Parallel Machine Scheduling With Sequence Dependent Family Setups , 2021, IEEE Access.

[20]  Lenz Belzner,et al.  Optimization of global production scheduling with deep reinforcement learning , 2018 .

[21]  Thomas Bauernhansl,et al.  Production Scheduling in Complex Job Shops from an Industry 4.0 Perspective: A Review and Challenges in the Semiconductor Industry , 2016, SAMI@iKNOW.

[22]  L. Buşoniu,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[23]  Yazid Mati,et al.  Scheduling challenges and approaches in semiconductor manufacturing , 2006 .

[24]  John W. Fowler,et al.  Semiconductor Manufacturing Scheduling and Dispatching , 2006 .

[25]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[26]  R. Lewin,et al.  MASTERING THE GAME , 1998 .