Fundamental Entropic Laws and Lp Limitations of Feedback Systems: Implications for Machine-Learning-in-the-Loop Control

In this paper, we study the fundamental performance limitations for generic feedback systems in which both the controller and the plant may be arbitrarily causal while the disturbance can be with any distributions. In particular, we obtain fundamental $\mathcal{L}_p$ bounds based on the entropic laws that are inherent in any feedback systems. We also examine the implications of the generic bounds for machine-learning-in-the-loop control; in other words, fundamental limits in general exist to what machine learning elements in feedback loops can achieve.

[1]  K. Åström.  Introduction to Stochastic Control Theory , 1970 .

[2]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[3]  Jie Chen,et al.  Power Gain Bounds of MIMO Networked Control Systems: An Entropy Perspective , 2019, IEEE Transactions on Automatic Control.

[4]  Jie Chen,et al.  Design constraints and limits of networked feedback in disturbance attenuation: An information-theoretic analysis , 2017, Autom..

[5]  Richard S. Sutton,et al.  Neural networks for control , 1990 .

[6]  Pablo A. Iglesias,et al.  Nonlinear extension of Bode's integral based on an information-theoretic interpretation , 2003, Syst. Control. Lett..

[7]  Frank L. Lewis,et al.  Optimal and Autonomous Control Using Reinforcement Learning: A Survey , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Frank L. Lewis,et al.  Reinforcement learning and optimal adaptive control: An overview and implementation examples , 2012, Annu. Rev. Control..

[9]  Jie Chen,et al.  Towards Integrating Control and Information Theories , 2017 .

[10]  Juš Kocijan,et al.  Modelling and Control of Dynamic Systems Using Gaussian Process Models , 2015 .

[11]  Thomas Parisini,et al.  Neural Approximations for Optimal Control and Decision , 2019 .

[12]  Jie Chen,et al.  Fundamental error bounds in state estimation: An information-theoretic analysis , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[13]  S. Dolinar,et al.  Maximum-entropy probability distributions under Lp-norm constraints , 1991 .

[14]  Jie Chen,et al.  Tradeoffs in Networked Feedback Systems: From Information-Theoretic Measures to Bode-Type Integrals , 2017, IEEE Transactions on Automatic Control.

[15]  Richard S. Sutton,et al.  Reinforcement Learning is Direct Adaptive Optimal Control , 1992, 1991 American Control Conference.

[16]  H. W. Bode,et al.  Network analysis and feedback amplifier design , 1945 .

[17]  G. Folland,et al.  The uncertainty principle: A mathematical survey , 1997 .

[18]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[19]  Munther A. Dahleh,et al.  Fundamental Limitations of Disturbance Attenuation in the Presence of Side Information , 2005, CDC 2005.

[20]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[21]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[22]  Masashi Sugiyama,et al.  Statistical Reinforcement Learning - Modern Machine Learning Approaches , 2015, Chapman and Hall / CRC machine learning and pattern recognition series.

[23]  Pieter Abbeel,et al.  Benchmarking Deep Reinforcement Learning for Continuous Control , 2016, ICML.

[24]  Steven L. Brunton,et al.  Machine Learning Control – Taming Nonlinear Dynamics and Turbulence , 2016, Fluid Mechanics and Its Applications.

[25]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[26]  N. Wiener,et al.  The prediction theory of multivariate stochastic processes , 1957 .

[27]  R. Gray.  Entropy and Information Theory , 1990, Springer New York.

[28]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[29]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[30]  D. Rajan.  Probability, Random Variables, and Stochastic Processes , 2017 .

[31]  Lucian Busoniu,et al.  Reinforcement learning for control: Performance, stability, and deep approximators , 2018, Annu. Rev. Control..

[32]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[33]  Shinji Hara,et al.  Achievable sensitivity bounds for MIMO control systems via an information theoretic approach , 2011, Syst. Control. Lett..

[34]  Ronald Davis,et al.  Neural networks and deep learning , 2017 .

[35]  Shinji Hara,et al.  Characterization of a complementary sensitivity property in feedback control: An information theoretic approach , 2009, Autom..

[36]  Benjamin Recht,et al.  A Tour of Reinforcement Learning: The View from Continuous Control , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[37]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[38]  Munther A. Dahleh,et al.  Feedback Control in the Presence of Noisy Channels: “Bode-Like” Fundamental Limitations of Performance , 2008, IEEE Transactions on Automatic Control.

[39]  F. Lewis,et al.  Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers , 2012, IEEE Control Systems.

[40]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.