Double Deep-Q Learning-Based Output Tracking of Probabilistic Boolean Control Networks

In this article, a reinforcement learning (RL)-based scalable technique is presented to control the probabilistic Boolean control networks (PBCNs). In particular, a double deep-<inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (DD<inline-formula> <tex-math notation="LaTeX">$Q\text{N}$ </tex-math></inline-formula>) approach is firstly proposed to address the output tracking problem of PBCNs, and optimal state feedback controllers are obtained such that the output of PBCNs tracks a constant as well as a time-varying reference signal. The presented method is model-free and offers scalability, thereby provides an efficient way to control large-scale PBCNs that are a natural choice to model gene regulatory networks (GRNs). Finally, three PBCN models of GRNs including a 16-gene and 28-gene networks are considered to verify the presented results.

[1]  A. Datta,et al.  On approximate stochastic control in genetic regulatory networks. , 2007, IET systems biology.

[2]  Yi Zhang,et al.  Human-like Autonomous Vehicle Speed Control by Deep Reinforcement Learning with Double Q-Learning , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[3]  Hado van Hasselt,et al.  Double Q-learning , 2010, NIPS.

[4]  Yuzhen Wang,et al.  State feedback based output tracking control of probabilistic Boolean networks , 2016, Inf. Sci..

[5]  Yang Liu,et al.  Robust Control Invariance of Probabilistic Boolean Control Networks via Event-Triggered Control , 2018, IEEE Access.

[6]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[7]  Jinde Cao,et al.  Asymptotic Output Tracking of Probabilistic Boolean Control Networks , 2020, IEEE Transactions on Circuits and Systems I: Regular Papers.

[8]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[9]  Daniel S. Kermany,et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning , 2018, Cell.

[10]  Lucian Busoniu,et al.  Reinforcement learning for control: Performance, stability, and deep approximators , 2018, Annu. Rev. Control..

[11]  Zhen Ni,et al.  A Multistage Game in Smart Grid Security: A Reinforcement Learning Solution , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Karl Henrik Johansson,et al.  Efficient Verification of Observability and Reconstructibility for Large Boolean Control Networks With Special Structures , 2020, IEEE Transactions on Automatic Control.

[13]  Stefan Preitl,et al.  Iterative Feedback and Learning Control. Servo systems applications , 2007 .

[14]  Baopu Li,et al.  A Control Strategy of Autonomous Vehicles Based on Deep Reinforcement Learning , 2016, 2016 9th International Symposium on Computational Intelligence and Design (ISCID).

[15]  Mobyen Uddin Ahmed,et al.  A Machine Learning Approach to Classify Pedestrians’ Event based on IMU and GPS , 2019 .

[16]  Edward R. Dougherty,et al.  Probabilistic Boolean networks: a rule-based uncertainty model for gene regulatory networks , 2002, Bioinform..

[17]  D. Cheng,et al.  Analysis and control of Boolean networks: A semi-tensor product approach , 2010, 2009 7th Asian Control Conference.

[18]  Sotiris Moschoyiannis,et al.  Deep Reinforcement Learning for Control of Probabilistic Boolean Networks , 2019, ArXiv.

[19]  Richard S. Sutton,et al.  Weighted importance sampling for off-policy learning with linear function approximation , 2014, NIPS.

[20]  Qiang Wei,et al.  Synchronization of Coupled Boolean Networks With Different Update Scheme , 2020, IEEE Access.

[21]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[22]  Luigi Glielmo,et al.  Output Tracking Control of Probabilistic Boolean Control Networks , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[23]  Daizhan Cheng,et al.  A Linear Representation of Dynamics of Boolean Networks , 2010, IEEE Transactions on Automatic Control.

[24]  Yang Liu,et al.  Output tracking of probabilistic Boolean networks by output feedback control , 2019, Inf. Sci..

[25]  Tielong Shen,et al.  A Finite Convergence Criterion for the Discounted Optimal Control of Stochastic Logical Networks , 2018, IEEE Transactions on Automatic Control.

[26]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[27]  Fuad E. Alsaadi,et al.  Output Regulation of Boolean Control Networks With Nonuniform Sampled-Data Control , 2019, IEEE Access.

[28]  Luigi Glielmo,et al.  Reachability and Controllability of Delayed Switched Boolean Control Networks , 2018, 2018 European Control Conference (ECC).

[29]  Yang Liu,et al.  Set Stability and Stabilization of Switched Boolean Networks With State-Based Switching , 2018, IEEE Access.

[30]  Jianquan Lu,et al.  Output Tracking of Boolean Control Networks Driven by Constant Reference Signal , 2019, IEEE Access.

[31]  Radu-Emil Precup,et al.  Hybrid data-driven fuzzy active disturbance rejection control for tower crane systems , 2020, Eur. J. Control.

[32]  Mauricio Barahona,et al.  Toggling a Genetic Switch Using Reinforcement Learning , 2013, ArXiv.

[33]  Aniruddha Datta,et al.  Optimal Intervention Strategies for Cyclic Therapeutic Methods , 2009, IEEE Transactions on Biomedical Engineering.

[34]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[35]  Mauricio Barahona,et al.  On periodic reference tracking using batch-mode reinforcement learning with application to gene regulatory network control , 2013, 52nd IEEE Conference on Decision and Control.

[36]  Weihua Gui,et al.  Asymptotical Stability of Logic Dynamical Systems With Random Impulsive Disturbances , 2021, IEEE Transactions on Automatic Control.

[37]  Tong Heng Lee,et al.  Infinite-Horizon Optimal Control of Switched Boolean Control Networks With Average Cost: An Efficient Graph-Theoretical Approach , 2019, IEEE Transactions on Cybernetics.

[38]  Luigi Glielmo,et al.  Reinforcement Learning Approach to Feedback Stabilization Problem of Probabilistic Boolean Control Networks , 2020, IEEE Control Systems Letters.

[39]  Ettore Fornasini,et al.  Optimal Control of Boolean Control Networks , 2014, IEEE Transactions on Automatic Control.

[40]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[41]  Yishay Mansour,et al.  Learning Rates for Q-learning , 2004, J. Mach. Learn. Res..

[42]  Assieh Saadatpour,et al.  Boolean modeling of biological regulatory networks: a methodology tutorial. , 2013, Methods.

[43]  Jing Wang,et al.  Finite-Time Controllability and Set Controllability of Impulsive Probabilistic Boolean Control Networks , 2020, IEEE Access.

[44]  Witold Pedrycz,et al.  Genetic learning of fuzzy cognitive maps , 2005, Fuzzy Sets Syst..

[45]  Hong-Bin Shen,et al.  Predicting gene regulatory interactions based on spatial gene expression data and deep learning , 2019, PLoS Comput. Biol..

[46]  Long Ji Lin,et al.  Self-improving reactive agents based on reinforcement learning, planning and teaching , 1992, Machine Learning.

[47]  Rui-Sheng Wang,et al.  Boolean modeling in systems biology: an overview of methodology and applications , 2012, Physical biology.

[48]  Ettore Fornasini,et al.  Observability and Reconstructibility of Probabilistic Boolean Networks , 2020, IEEE Control Systems Letters.

[49]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[50]  Luigi Glielmo,et al.  Feedback stabilization control design for switched Boolean control networks , 2020, Autom..

[51]  Luigi Glielmo,et al.  Equilibrium and stability analysis of X-chromosome linked recessive diseases model , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).