Simulation vs . Emulation : Evaluating Mobile Ad Hoc Network Routing Protocols

In order for simulation studies to be useful, it is very important that the simulation results match as closely as possible with the testbed results. This paper compares emulated testbed results with simulation results from NS2 and GloMoSim. OLSR was used as a routing protocol and NRL Mobile Network Emulator (MNE) for dynamic topology control and manipulation. Five Linux based laptops, equipped with IEEE 802.11b wireless network cards were used for testbed implementation. At low traffic rates, testbed results matched closely with the simulation results, at higher traffic rates, testbed results not only differed from the simulation results both qualitatively and quantitatively but the simulation results from both the simulators were barely comparable in some scenarios. Introduction Network engineers use simulation tools to debug and test the reliability and QoS of network protocols as well as hardware equipment. This makes simulation a very important step towards the deployment of wireless communication networks. A simulation is only useful if the simulation results match as closely as possible with the testbed results. With the popularity of commercial wireless devices such as IEEE 802.11 and Bluetooth, a transition is taking place from traditional wired networks to wireless networks. In the wireless domain, Mobile Ad-hoc networks are getting more and more popular due to their flexible, dynamic, self-healing and self-configuring nature. In spite all the technological achievements and cutting edge research occurring in the field of mobile wireless networks, there are growing concerns regarding the reliability of results generated by wireless network simulators. Related Work There are very few papers discussing the accuracy of simulation results and the variation of the results among different wireless network simulators. An interesting paper published by Distributed Systems Laboratory, Ecole Polytechnique Fédérale de Lausanne, compares the simulation results of OPNET, NS2 and GloMoSim. IEEE 802.11 MAC implementations and a very simple flooding algorithm were used to simulate wireless ad-hoc networks using 50 mobile nodes. According to the paper, there was huge diversion between results from different simulators and in some cases results were barely comparable. The paper also mentioned that there is a scarcity of real experiments that demonstrate the correctnes of wireless network simulators [1]. This paper gave us our primary motivation to perform simulations using various simulation tools available and compare the simulation results with the testbed results. Kunz compared MANET simulation results with testbed results using the BCAST protocol, a multicast routing protocol. According to his findings, in a stationary environment, all results were close to each other, and a mobile emulator introduces negligible additional overhead. However, in mobile scenarios, his measurements in the emulated environment differed both qualitatively and quantitatively from the simulation results. According to the report, tools like MNE (Mobile Network Emulator) can help in testing and debugging protocols for real testbed but it is not clear if these tools are suitable for the performance evaluation of the protocol [2]. Based on the findings of the two above research projects, a need for detailed and carefully planed simulation and emulation experiments was realized. To ensure the reliability of the results, it was decided to use different traffic models and a highly dynamic network topology. The primary goal of the project was to find out where simulation and emulation yield drastically different results and come up with guidelines for how/whether to use simulation as an appropriate tool for performance studies of protocol. OLSR Protocol The Optimized Link State Routing protocol (RFC 3626) is a proactive routing protocol, which uses a link-state scheme in an optimized manner to diffuse topology information. OLSR message flooding is optimized to run in a multi-hop wireless network to preserve bandwidth. “Optimization is acquired through a technique called MultiPoint Relaying. The idea of MultiPoint Relaying (MPR) is to minimize the overhead of flooding messages by restricting the control messages to selected nodes.” [3] MNE MNE is designed to work under Linux-based distribution environments with IPTABLES network filtering capabilities installed. A packet filter looks at the header of packets as they pass through and based on user-defined rules decides whether the packet should be accepted for further processing or should be dropped. Packet filtering is used for security, control and watchfulness. MNE takes advantage of these control features of IPTABLES to block packets from a particular node. The blocking is done based on the wireless link range. In MNE, packets from the nodes that are deemed out of range are blocked with the IPTABLES MAC Filtering component [4]. The control module provides a communication back channel for transmitting the node’s location information. The location update information is constantly transferred to all the nodes in the network using a broadcast channel (also known as back channel or controlled network). Either the wired or the wireless interface can be used to transmit location updates; however transmitting both traffics (dynamic topology information and user traffic) on the same physical channel causes congestion. In our work, the wired interface was used to broadcast location update to avoid congestion on the wireless channel. MNE supports several different ways of emulating node motion, which includes file, random, ns2, static, line and circle. In MNE, NS2 motion files can be used directly. The MOVE module of MNE provides location information represented in terms of longitude and latitude and this information is then transmitted to all the nodes through the controlled network described above. Each node in the network broadcasts its location information with the MAC address of the wireless network card over the back channel. The DYNAMICS module uses this information for calculating the distance between the nodes and performing filtering. The MAC address is used to block packets from a particular node. Dynamic Network Topology Dynamic network topology scenarios were carefully selected to simulate 1 hop, 2 hop and multi-hop connections in different scenarios. An area of 600x600 meters was selected as the basis for our dynamic topology. During the simulation of 400 seconds, the nodes acquired four different topologies. (a) Scenario 1 (b) Scenario 2 (c) Scenario 3 (d) Scenario 4 Figure 1: Network Topologies The above figure shows the orientation of nodes at time t = 0, 180, 300 and 400 seconds. In first scenario at time t = 0, nodes n0 and n1 are within each other’s direct communication range, n4 and n3 can also send and receive packets but n2 is out of range. Shortly after t = 0 sec, all the nodes start moving and at t = 180 sec they acquire positions shown in scenario 2. In scenario 2 node n1 cannot communicate with n4 directly and has to go through an intermediate node and vice versa, hence establishing a 2-hop communication network. In this scenario any node (n0, n2, n3) can be chosen as a MPR to relay all the packets. Scenario 3 and 4 were designed to simulate multi hop communication. Simulation Model NS2 [5] is a discrete event simulator for, among others, Linux that can be used to define different scenarios. NS2 depends on several external components such as Tcl, otcl and TclCL. A simulation is defined by a TCL program and a topology is defined using ns commands. NS2 can be used to simulate wired or wireless network. NS2 includes a model for IEEE 802.11 and the protocol stack includes implementations for ad-hoc routing protocols which makes it an ideal simulator for mobile ad hoc networks. GloMoSim is a scalable simulation environment for wireless and wired network systems designed by UCLA’s parallel computing laboratory [6]. GloMoSim can be used to simulate a variety of wireless network protocols. The protocol stack includes models for the channel, radio, MAC, network, transport, and higher layers. For both simulators, different research groups have developed OLSR models and made them publicly available. We integrated these models into NS2 and GloMoSim and ensured that we used consistent values for all protocol parameters (such as the periodicity of HELLO messages). Five mobile nodes were used in the simulation model. The MAC and the Logic Link Layer implemented in NS2 and GloMoSim for IEEE 802.11 was used in the simulation. A Drop Tail Queue of size 100 packets was selected as an interface queue. An omni-direction antenna having unity gain was used. A two-ray ground reflection model was used for radio propagation. Traffic Model CBR agents were used to generate traffic in the network. The following table shows the packet size and number of packets sent per second for all the experiments. Experiment # Packet Size Duration 1 128 bytes 1 packet/sec 2 128 bytes 10 packets/sec 3 512 bytes 10 packets/sec Table 1: Traffic Rates Each node’s CBR was connected to every other nodes receiver agent hence forming a meshed network. All the links were bi-directional. User Datagram Protocol (UDP) was used as Transport Layer protocol to minimise the overhead of establishing a connection. Testbed Implementation NRL MNE was used for dynamic topology control and manipulation. The Ethernet wired interface was used to broadcast location update to avoid congestion on the wireless channel. An implementation of OLSR (RFC 3626) from the PROTocol Engineering Advanced Networking (PROTEAN) Research Group was used for the testbed experiments. Multi-Generator (MGEN version 4) (an open source software from NRL) was used to generate traffic over the wireless channel to obtain packet delivery statistics. MGEN provides the ability to perform IP network performance tests and measurements using UDP/IP traffic. To ensure the reliabi