MANET protocol simulations considered harmful: the case for benchmarking

In this article, we investigate the current best practices in simulation-based multi-hop wireless ad-hoc network (MANET) protocol evaluation. We extend a prior characterization of the settings and parameters used in MANET simulations by studying the papers published in one of the premier mobile networking conferences between 2006 and 2010. We find that there are still several configuration pitfalls which many papers fall victim to, which in turn damages the integrity of the results as well as any research aimed at reproducing and extending these results. We then describe the simulation "design space" of MANET routing in terms of its basic dimensions and corresponding parameters. We also propose four "auxiliary" metrics to increase simulation integrity. We conclude with several example scenarios that promote modeling simulations after real-world situations.

[1]  Ivan Stojmenovic.  Simulations in wireless sensor and ad hoc networks: matching and advancing models, metrics, and solutions , 2008, IEEE Communications Magazine.

[2]  Tracy Camp,et al.  MANET simulation studies: the incredibles , 2005, MOCO.

[3]  T. Kunz,et al.  Towards network simulations credibility: Lessons from applying five key principles , 2008, MILCOM 2008 - 2008 IEEE Military Communications Conference.

[4]  Ian Wakeman,et al.  The state of peer-to-peer simulators and simulations , 2007, CCRV.

[5]  Todd R. Andel,et al.  On the credibility of manet simulations , 2006, Computer.

[6]  Katia Obraczka,et al.  On the Invariance of Spatial Node Density for Realistic Mobility Modeling , 2011, 2011 IEEE Eighth International Conference on Mobile Ad-Hoc and Sensor Systems.

[7]  Tracy Camp,et al.  Constructing rigorous MANET simulation scenarios with realistic mobility , 2010, 2010 European Wireless Conference (EW).

[8]  Tracy Camp,et al.  Two Standards for Rigorous MANET Routing Protocol Evaluation , 2006, 2006 IEEE International Conference on Mobile Ad Hoc and Sensor Systems.

[9]  Mingyan Liu,et al.  Random waypoint considered harmful , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[10]  Thomas Fuhrmann,et al.  Towards Comparable Network Simulations , 2008 .