Effectiveness of Multi-device Testing Mobile Applications

The paper evaluates methods of selecting mobile devices (i.e., Smartphones and tablets) to test mobile software applications. Due to the number of such devices on the market and the variations in their characteristics, it is hard to guarantee that an application will work properly on all devices. Software faults found only in certain devices are known as device-specific faults. The goal of this research was to determine how many devices must be tested and which methods of device selection are best for revealing these device-specific faults. To collect the experimental data, 15 Android applications were tested on 30 mobile devices using real devices and remote testing services such as Test Object, pCloudy, and others. In total, 24 device-specific faults were detected. Using our data, we analyzed the different methods of selection based on the characteristics of each device. Our research showed that a random selection of 13 devices achieved 100% effectiveness. However, coverage of device characteristics in the selection process enables the achievement of an acceptable 90% level of effectiveness with sets of only five devices. The most successful approach was the coverage of different types of Android operating systems. Our results include recommendations that will help increase the effectiveness and decrease the costs of mobile testing.