Joint control of manufacturing and onsite microgrid system via novel neural-network integrated reinforcement learning algorithms

Microgrid is a promising technology of distributed energy supply system, which typically consists of storage devices, generation capacities including renewable sources, and controllable loads. It has been widely investigated and applied for residential & commercial end use customers as well as critical facilities. In this paper, we propose a joint dynamic control model of microgrids and manufacturing systems using Markov Decision Process (MDP) to identify an optimal control strategy for both microgrid components and manufacturing system so that the energy cost for production can be minimized without sacrificing production throughput. The proposed MDP model has a high dimensional state/action space and is complicated in that the state and action spaces have both discrete and continuous parts and are intertwined through constraints. To resolve these challenges, a novel reinforcement learning algorithm that leverages both on-policy temporal difference control (TDcontrol) and deterministic policy gradient (DPG) algorithms is proposed. In this algorithm, the values of discrete decision actions are learned through neural network integrated temporal difference iteration, while the parameterized values of continuous actions are learned from deterministic policy gradients. The constraints are then addressed via proximal projection operators at the policy gradient updates. Experiments for a manufacturing system with an onsite microgrid with renewable sources have been implemented to identify optimal control actions for both manufacturing system and microgrid components towards cost optimality. The experimental results show the effectiveness of combining TD control and policy gradient methodologies in addressing the “curse of dimensionality” in dynamic decisionmaking with high dimensional and complicated state and action spaces.