Data Collection in Sensor Networks via the Novel Fast Markov Decision Process Framework

We investigate the data collection problem in sensor networks. The network consists of a number of stationary sensors deployed at different sites for sensing and storing data locally. A mobile element moves from site to site to collect data from the sensors periodically. There are different costs associated with the mobile element moving from one site to another, and different rewards for obtaining data at different sensors. Furthermore, the costs and the rewards are assumed to change abruptly. The goal is to find a “fast” optimal movement pattern/policy of the mobile element that optimizes for the costs and rewards in non-stationary environments. We formulate and solve this problem using a novel optimization framework called fast Markov decision process (FMDP). The proposed FMDP framework extends the classical Markov decision process theory by incorporating the notion of mixing time that allows for the trade-off between the optimality and the convergence rate to the optimality of a policy. Theoretical and simulation results are provided to verify the proposed approach.

[1]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[2]  Elizabeth L. Wilmer,et al.  Markov Chains and Mixing Times , 2008 .

[3]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[4]  Stephen P. Boyd,et al.  Fastest Mixing Markov Chain on a Graph , 2004, SIAM Rev..

[5]  Ashutosh Sabharwal,et al.  Using Predictable Observer Mobility for Power Efficient Design of Sensor Networks , 2003, IPSN.

[6]  Sajal K. Das,et al.  Data Collection in Wireless Sensor Networks with Mobile Elements: A Survey , 2011, TOSN.

[7]  Waylon Brunette,et al.  Data MULEs: modeling and analysis of a three-tier architecture for sparse sensor networks , 2003, Ad Hoc Networks.

[8]  Ellen W. Zegura,et al.  A message ferrying approach for data delivery in sparse mobile ad hoc networks , 2004, MobiHoc '04.

[9]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[10]  John Odentrantz,et al.  Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues , 2000, Technometrics.

[11]  Donghyun Kim,et al.  Minimizing data collection latency in wireless sensor network with multiple mobile elements , 2012, 2012 Proceedings IEEE INFOCOM.

[12]  Gaetano Borriello,et al.  Exploiting Mobility for Energy Efficient Data Collection in Wireless Sensor Networks , 2006, Mob. Networks Appl..

[13]  Rajesh K. Gupta,et al.  Optimal Speed Control of Mobile Node for Data Collection in Sensor Networks , 2010, IEEE Transactions on Mobile Computing.

[14]  Ellen W. Zegura,et al.  Power management in delay tolerant networks: a framework and knowledge-based mechanisms , 2005, 2005 Second Annual IEEE Communications Society Conference on Sensor and Ad Hoc Communications and Networks, 2005. IEEE SECON 2005..

[15]  Thomas F. La Porta,et al.  Sensor relocation in mobile sensor networks , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[16]  Yu-Chee Tseng,et al.  iMouse: An Integrated Mobile Surveillance and Wireless Sensor System , 2007, Computer.

[17]  Yuanyuan Yang,et al.  SenCar: An Energy-Efficient Data Gathering Mechanism for Large-Scale Multihop Sensor Networks , 2006, IEEE Transactions on Parallel and Distributed Systems.

[18]  David J. Aldous,et al.  Lower bounds for covering times for reversible Markov chains and random walks on graphs , 1989 .

[19]  Guoliang Xing,et al.  Rendezvous Planning in Wireless Sensor Networks with Mobile Elements , 2008, IEEE Transactions on Mobile Computing.

[20]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[21]  Thinh P. Nguyen,et al.  Adiabatic Markov Decision Process with application to queuing systems , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[22]  Leonidas Georgiadis,et al.  Energy-aware Routing to Maximize Lifetime in Wireless Sensor Networks with Mobile Sink , 2017 .

[23]  Subir Biswas,et al.  Joint routing and navigation protocols for data harvesting in sensor networks , 2008, 2008 5th IEEE International Conference on Mobile Ad Hoc and Sensor Systems.

[24]  R. Bellman A Markovian Decision Process , 1957 .

[25]  G. Monahan State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms , 1982 .

[26]  Gaurav S. Sukhatme,et al.  Achieving connectivity through coalescence in mobile robot networks , 2007 .

[27]  Thinh P. Nguyen,et al.  Fast Markov Decision Process for data collection in sensor networks , 2014, 2014 23rd International Conference on Computer Communication and Networks (ICCCN).

[28]  Tetsuro Morimura,et al.  Mixing-Time Regularized Policy Gradient , 2014, AAAI.

[29]  K DasSajal,et al.  Data Collection in Wireless Sensor Networks with Mobile Elements , 2011 .

[30]  Thinh P. Nguyen,et al.  Network Protocol Designs: Fast Queuing Policies via Convex Relaxation , 2014, IEEE Transactions on Communications.

[31]  Zygmunt J. Haas,et al.  A new networking model for biological applications of ad hoc sensor networks , 2006, TNET.

[32]  Giuseppe Anastasi,et al.  A Hybrid Adaptive Protocol for Reliable Data Delivery in WSNs with Multiple Mobile Sinks , 2011, Comput. J..

[33]  Mounir Torki,et al.  Second-order directional derivatives of all eigenvalues of a symmetric matrix , 2001 .

[34]  Deborah Estrin,et al.  Intelligent fluid infrastructure for embedded networks , 2004, MobiSys '04.

[35]  R. Bellman,et al.  Dynamic Programming and Markov Processes , 1960 .

[36]  Yong Wang,et al.  Energy-efficient computing for wildlife tracking: design tradeoffs and early experiences with ZebraNet , 2002, ASPLOS X.

[37]  Peter L. Bartlett,et al.  Infinite-Horizon Policy-Gradient Estimation , 2001, J. Artif. Intell. Res..

[38]  A SomasundaraA.,et al.  Mobile Element Scheduling for Efficient Data Collection in Wireless Sensor Networks with Dynamic Deadlines , 2004 .

[39]  Chang-Gun Lee,et al.  Partitioning based mobile element scheduling in wireless sensor networks , 2005, 2005 Second Annual IEEE Communications Society Conference on Sensor and Ad Hoc Communications and Networks, 2005. IEEE SECON 2005..