Optimal energy allocation for linear control with packet loss under energy harvesting constraints

This paper studies a closed loop linear control system over a lossy communication link. A sensor computes a state estimate of the observed discrete-time system and sends it (in the form of packetized transmission) to the controller in the receiver block over a randomly time-varying (fading) packet dropping link. The receiver sends an ACK/NACK packet to the transmitter over an acknowledgement channel which might also be prone to packet loss. It is assumed that the energy used in packet transmission depletes a battery of limited capacity at the sensor, but is also replenished by an energy harvester which has access to a source of everlasting but random harvested energy. Under an assumption of finite-state Markov chain models of the energy harvesting and the fading channel gain processes, the objective is to design an optimal energy allocation policy at the transmitter and an optimal control policy at the receiver so that an average infinite horizon linear quadratic Gaussian (LQG) control cost is minimized. It is shown that in the case of perfect channel feedback a separation principle holds, the optimal LQG controller is linear and the optimal energy allocation policy at the transmitter can be obtained via solving the Bellman dynamic programming equation. A Q-learning algorithm is used to approximate the optimal energy allocation policy in case the system parameters are unknown. Numerical simulation examples show that the dynamic programming based policies outperform various simple heuristic policies, especially at higher battery capacities.

[1]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[2]  Subhrakanti Dey,et al.  Dynamic Quantizer Design for Hidden Markov State Estimation Via Multiple Sensors With Fusion Center Feedback , 2006, IEEE Transactions on Signal Processing.

[3]  J.P. Hespanha,et al.  Estimation under uncontrolled and controlled communications in Networked Control Systems , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[4]  Subhrakanti Dey,et al.  Optimal Energy Allocation for Kalman Filtering Over Packet Dropping Links With Imperfect Acknowledgments and Energy Harvesting Constraints , 2014, IEEE Transactions on Automatic Control.

[5]  Daniel E. Quevedo,et al.  An Optimal Transmission Strategy for Kalman Filtering Over Packet Dropping Links With Imperfect Acknowledgements , 2014, IEEE Transactions on Control of Network Systems.

[6]  Bruno Sinopoli,et al.  Foundations of Control and Estimation Over Lossy Networks , 2007, Proceedings of the IEEE.

[7]  Daniel E. Quevedo,et al.  On Kalman filtering over fading wireless channels with controlled transmission powers , 2012, Autom..

[8]  Chee-Yee Chong,et al.  Sensor networks: evolution, opportunities, and challenges , 2003, Proc. IEEE.

[9]  Subhrakanti Dey,et al.  Stability of Kalman filtering with Markovian packet losses , 2007, Autom..

[10]  Sandra Hirche,et al.  On the Optimality of Certainty Equivalence for Event-Triggered Control Systems , 2013, IEEE Transactions on Automatic Control.

[11]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[12]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[13]  Rui Zhang,et al.  Optimal Energy Allocation for Wireless Communications With Energy Harvesting Constraints , 2011, IEEE Transactions on Signal Processing.

[14]  Alessandro Chiuso,et al.  Linear encoder-decoder-controller design over channels with packet loss and quantization noise , 2015, 2015 European Control Conference (ECC).

[15]  Daniel E. Quevedo,et al.  State Estimation Over Sensor Networks With Correlated Wireless Fading Channels , 2013, IEEE Transactions on Automatic Control.

[16]  S. Sastry,et al.  An LQG Optimal Linear Controller for Control Systems with Packet Losses , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[17]  Mircea Lazar,et al.  Event Based State Estimation With Time Synchronous Updates , 2012, IEEE Transactions on Automatic Control.

[18]  S. Sastry,et al.  Optimal control with unreliable communication: the TCP case , 2005, Proceedings of the 2005, American Control Conference, 2005..

[19]  Jing Yang,et al.  Transmission with Energy Harvesting Nodes in Fading Wireless Channels: Optimal Policies , 2011, IEEE Journal on Selected Areas in Communications.

[20]  Gerhard P. Hancke,et al.  Industrial Wireless Sensor Networks: Challenges, Design Principles, and Technical Approaches , 2009, IEEE Transactions on Industrial Electronics.

[21]  A. Goldsmith,et al.  Kalman filtering with partial observation losses , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[22]  K. Åström,et al.  Comparison of Riemann and Lebesgue sampling for first order stochastic systems , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[23]  Gerhard P. Hancke,et al.  Opportunities and Challenges of Wireless Sensor Networks in Smart Grid , 2010, IEEE Transactions on Industrial Electronics.

[24]  K. Poolla,et al.  Time varying optimal control with packet losses , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[25]  Richard M. Murray,et al.  Optimal LQG control across packet-dropping links , 2007, Syst. Control. Lett..

[26]  Manfred Schäl,et al.  Average Optimality in Dynamic Programming with General State Space , 1993, Math. Oper. Res..

[27]  Ling Shi,et al.  Sensor data scheduling for optimal state estimation with communication energy constraint , 2011, Autom..

[28]  Luca Schenato,et al.  Optimal Estimation in Networked Control Systems Subject to Random Delay and Packet Drop , 2008, IEEE Transactions on Automatic Control.

[29]  Vinod Sharma,et al.  Optimal energy management policies for energy harvesting sensor nodes , 2008, IEEE Transactions on Wireless Communications.

[30]  Jing Yang,et al.  Broadcasting with an Energy Harvesting Rechargeable Transmitter , 2010, IEEE Transactions on Wireless Communications.

[31]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[32]  Chin Keong Ho,et al.  Markovian models for harvested energy in wireless communications , 2010, 2010 IEEE International Conference on Communication Systems.

[33]  Bruno Sinopoli,et al.  LQG CONTROL WITH MISSING OBSERVATION AND CONTROL PACKETS , 2005 .

[34]  P. Sadeghi,et al.  Finite-state Markov modeling of fading channels - a survey of principles and applications , 2008, IEEE Signal Processing Magazine.

[35]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[36]  John G. Proakis,et al.  Digital Communications , 1983 .

[37]  Daniel E. Quevedo,et al.  Energy Efficient State Estimation With Wireless Sensors Through the Use of Predictive Power Control and Coding , 2010, IEEE Transactions on Signal Processing.

[38]  Bruno Sinopoli,et al.  A characterization of the critical value for Kalman filtering with intermittent observations , 2008, 2008 47th IEEE Conference on Decision and Control.

[39]  Aylin Yener,et al.  Optimum Transmission Policies for Battery Limited Energy Harvesting Nodes , 2010, IEEE Transactions on Wireless Communications.

[40]  Ling Shi,et al.  Probabilistic performance of state estimation across a lossy network , 2008, Autom..

[41]  Ling Shi,et al.  Optimal DoS Attack Scheduling in Wireless Networked Control System , 2016, IEEE Transactions on Control Systems Technology.

[42]  Ian F. Akyildiz,et al.  Sensor Networks , 2002, Encyclopedia of GIS.

[43]  Bruno Sinopoli,et al.  Kalman filtering with intermittent observations , 2004, IEEE Transactions on Automatic Control.