Towards the Systematic Testing of Virtual Reality Programs

Background: Software testing is a critical activity to ensure that software complies with its specification. However, current software testing activities tend not to be completely effective when applied in specific software domains in Virtual Reality (VR) that has several new types of features such as images, sounds, videos, and differentiated interaction, which can become sources of new kinds of faults. Aims: This paper presents an overview of the main VR characteristics that can have an impact on verification, validation, and testing (VV&T). Furthermore, it analyzes some of the most successful VR open-source projects to draw a picture concerning the danger of the lack of software testing activities. Method: We compared the current state of software testing practice in open-source VR projects and evaluate how the lack of testing can be damaging to the development of a product. We assessed the incidence of code smells and verified how such projects behave concerning the tendency to present faults. We also perform the same analyses on projects that are not VR related to have a better understanding of these results. Results: The results showed that the practice of software testing is not yet widespread in the development of VR applications. It was also found that there is a high incidence of code smells in VR projects. Analyzing Non-VR projects we noticed that classes that have test cases tend to produce fewer smells compared to classes that were not tested. Regarding fault-proneness analysis, we used an unsupervised approach to VR and Non-VR projects. Results showed that about 12.2% of the classes analyzed in VR projects are fault-prone, while Non-VR projects presented a lower fault-proneness rate (8.9%). Conclusions: Regarding the application of software testing techniques on VR projects, it was observed that only a small number of projects are concerned about developing test cases for VR projects, perhaps because we still do not have the necessary tools to help in this direction. Concerning smells, we concluded that there is a high incidence in VR projects, especially regarding implementing smells and this high incidence can have a significant influence on faults. Finally, the study related to fault proneness pointed out that the lack of software testing activity is a significant risk to the success of the projects.

[1]  Martin Fowler,et al.  Refactoring - Improving the Design of Existing Code , 1999, Addison Wesley object technology series.

[2]  M. Fay,et al.  Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. , 2010, Statistics surveys.

[3]  Jun Yang,et al.  Defect Prediction on Unlabeled Datasets by Using Unsupervised Clustering , 2016, 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS).

[4]  Rafael Capilla,et al.  Software Architectures for Designing Virtual Reality Applications , 2004, EWSA.

[5]  Upulee Kanewala,et al.  Automated Test Oracles: State of the Art, Taxonomies, and Trends , 2015, Adv. Comput..

[6]  Gabriele Bavota,et al.  Do They Really Smell Bad? A Study on Developers' Perception of Bad Code Smells , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[7]  Yuanfang Cai,et al.  Hotspot Patterns: The Formal Definition and Automatic Detection of Architecture Smells , 2015, 2015 12th Working IEEE/IFIP Conference on Software Architecture.

[8]  Philippe Coiffet,et al.  Virtual Reality Technology , 2003, Presence: Teleoperators & Virtual Environments.

[9]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[10]  Yi Sun,et al.  Some Code Smells Have a Significant but Small Effect on Faults , 2014, TSEM.

[11]  Steven M. LaValle,et al.  Virtual Reality , 2004, Heilberufe.

[12]  Steffen Herbold,et al.  A systematic mapping study on cross-project defect prediction , 2017, ArXiv.

[13]  Xiao Liu,et al.  An empirical study on software defect prediction with a simplified metric set , 2014, Inf. Softw. Technol..

[14]  Daniela E. Damian,et al.  The promises and perils of mining GitHub , 2009, MSR 2014.

[15]  John K Haas,et al.  A History of the Unity Game Engine , 2014 .

[16]  Fred P. Brooks,et al.  The Mythical Man-Month , 1975, Reliable Software.

[17]  Tushar Sharma,et al.  Designite - A Software Design Quality Assessment Tool , 2016, 2016 IEEE/ACM 1st International Workshop on Bringing Architectural Design Thinking Into Developers' Daily Activities (BRIDGE).

[18]  Qinping Zhao,et al.  A survey on virtual reality , 2009, Science in China Series F: Information Sciences.

[19]  James M Hillenbrand,et al.  Open source software for experiment design and control. , 2005, Journal of speech, language, and hearing research : JSLHR.

[20]  Márcio Eduardo Delamaro,et al.  An automated functional testing approach for virtual reality applications , 2018, Softw. Test. Verification Reliab..

[21]  Ossi Taipale,et al.  Software Test Automation in Practice: Empirical Observations , 2010, Adv. Softw. Eng..

[22]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[23]  KarreSai Anirudh,et al.  Understanding usability evaluation setup for VR products in industry , 2020 .

[24]  Pooja Paramshetti,et al.  Survey on Software Defect Prediction Using Machine Learning Techniques , 2014 .

[25]  Diomidis Spinellis,et al.  House of Cards: Code Smells in Open-Source C# Repositories , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[26]  Antonia Bertolino,et al.  Do We Need New Strategies for Testing Systems-of-Systems? , 2018, 2018 IEEE/ACM 6th International Workshop on Software Engineering for Systems-of-Systems (SESoS).

[27]  Foutse Khomh,et al.  An exploratory study of the impact of antipatterns on class change- and fault-proneness , 2011, Empirical Software Engineering.

[28]  Carl Machover,et al.  Virtual reality , 1994, IEEE Computer Graphics and Applications.

[29]  Foutse Khomh,et al.  An Exploratory Study of the Impact of Code Smells on Software Change-proneness , 2009, 2009 16th Working Conference on Reverse Engineering.

[30]  Per Runeson,et al.  A survey of unit testing practices , 2006, IEEE Software.

[31]  Foutse Khomh,et al.  The State of Practice on Virtual Reality (VR) Applications: An Exploratory Study on Github and Stack Overflow , 2018, 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[32]  Xiaoyin Wang,et al.  An Empirical Study of Open Source Virtual Reality Software Projects , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[33]  Duane K. Boman,et al.  International Survey: Virtual-Environment Research , 1995, Computer.

[34]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[35]  Y. Raghu Reddy,et al.  Understanding usability evaluation setup for VR products in industry: a review study , 2020, SIAP.

[36]  Gabriele Bavota,et al.  An empirical investigation into the nature of test smells , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[37]  Tuukka M. Takala.  A Toolkit for Virtual Reality Software Development - Investigating Challenges, Developers, and Users ; Ohjelmistoalusta virtuaalitodellisuussovellusten kehitykseen: Katsaus haasteisiin, ohjelmistokehittäjiin, ja käyttäjiin , 2017 .

[38]  Gabriele Bavota,et al.  Code smells for Model-View-Controller architectures , 2017, Empirical Software Engineering.

[39]  Márcio Eduardo Delamaro,et al.  The Relationship between Requirements Engineering and Virtual Reality Systems: A Systematic Literature Review , 2013, 2013 XV Symposium on Virtual and Augmented Reality.

[40]  Neil Walkinshaw,et al.  Are 20% of files responsible for 80% of defects? , 2018, ESEM.

[41]  Gregory Tassey,et al.  Prepared for what , 2007 .

[42]  Girish Suryanarayana,et al.  Refactoring for Software Design Smells: Managing Technical Debt , 2014 .

[43]  A. Braga,et al.  The influence of time-to-market and target costing in the new product development success , 2008 .

[44]  Frederick P. Brooks,et al.  The Mythical Man-Month: Essays on Softw , 1978 .

[45]  Mauricio Finavaro Aniche,et al.  What Do the Asserts in a Unit Test Tell Us about Code Quality? A Study on Open Source and Industrial Projects , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[46]  Antonia Bertolino,et al.  Software Testing Research and Practice , 2003, Abstract State Machines.

[47]  Y. Raghu Reddy,et al.  Is Virtual Reality Product Development different?: An Empirical Study on VR Product Development Practices , 2019, ISEC.

[48]  Claudio Kirner,et al.  Fundamentos e Tecnologia de Realidade Virtual e Aumentada , 2006 .

[49]  Paolo Tonella,et al.  Capture-replay vs. programmable web testing: An empirical assessment during test case evolution , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).