What do game developers test in their products?

Context: Software projects often have four objectives; to produce the required functionality, in budget and in schedule, with acceptable quality. That statement may be true for most of the ordinary software development projects, but are these objectives enough for game development, where creativity and artistic aspects have a major role? Goal: We analyze how game developing organizations test their products, what are their main test objectives and how they perceive themselves in the software business. Method: We interviewed seven game development teams from different companies and studied how they test their products with grounded theory approach. Results: Our results suggest that game developers focus on soft values such as game content or user experience, instead of more traditional objectives such as reliability or efficiency. Conclusions: Game developers have similar, but not fully comparable to software industry, set of priorities in their software testing and quality assurance approaches.

[1]  Nahid Golafshani,et al.  Understanding Reliability and Validity in Qualitative Research , 2003 .

[2]  Monica McGill Defining the expectation gap: a comparison of industry needs and existing game development curriculum , 2009, FDG.

[3]  C. Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[4]  Naty Hoffman,et al.  Game development , 2006, SIGGRAPH Courses.

[5]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[6]  Jonathan Blow,et al.  Game Development: Harder Than You Think , 2004, ACM Queue.

[7]  Nathan H. Petschenik Practical Priorities in System Testing , 1985, IEEE Software.

[8]  Pamela Jordan Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[9]  Guy Paré,et al.  Using case study research to build theories of IT implementation , 1997 .

[10]  Marcelo Soares Pimenta,et al.  Houston, we have a problem...: a survey of actual problems in computer games development , 2008, SAC '08.

[11]  Barry W. Boehm,et al.  How Much Software Quality Investment Is Enough: A Value-Based Approach , 2006, IEEE Software.

[12]  Ossi Taipale,et al.  How Test Organizations Adopt New Testing Practices and Methods? , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[13]  Mayuram S. Krishnan,et al.  Evaluating the cost of software quality , 1998, CACM.

[14]  J. D. Roode,et al.  Glaserian and Straussian grounded theory: similar or completely different? , 2009, SAICSIT '09.

[15]  Kari Smolander,et al.  How Applicable Is ISO/IEC 29110 in Game Software Development? , 2013, PROFES.

[16]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[17]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[18]  C. Brodsky The Discovery of Grounded Theory: Strategies for Qualitative Research , 1968 .

[19]  S. A. Pedersen [Validity of qualitative research]. , 1990, Ugeskrift for laeger.

[20]  Andrea Bergmann,et al.  Testing Computer Software Second Edition , 2016 .

[21]  S. Murugesan Attitude towards testing: a key contributor to software quality , 1994, Proceedings of 1994 1st International Conference on Software Testing, Reliability and Quality Assurance (STRQA'94).

[22]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[23]  Mirva Peltoniemi,et al.  Life-cycle of the games industry: the specificities of creative industries , 2008, MindTrek '08.

[24]  Felix Redmill,et al.  Exploring risk‐based testing and its implications , 2004, Softw. Test. Verification Reliab..

[25]  Annakaisa Kultima,et al.  “Hopefully everything I'm doing has to do with innovation”: Games industry professionals on innovation in 2009 , 2010, 2010 2nd International IEEE Consumer Electronics Society's Games Innovations Conference.

[26]  Annakaisa Kultima The organic nature of game ideation: game ideas arise from solitude and mature by bouncing , 2010, Future Play.

[27]  Kevin A. Schneider,et al.  Requirements engineering and the creative process in the video game industry , 2005, 13th IEEE International Conference on Requirements Engineering (RE'05).

[28]  Ossi Taipale,et al.  Exploring the Perceived End-Product Quality in Software-Developing Organizations , 2012, Int. J. Inf. Syst. Model. Des..

[29]  Kari Smolander,et al.  What do game developers expect from development and design tools? , 2013, EASE '13.

[30]  Hisham M. Haddad,et al.  Software Engineering Challenges in Game Development , 2009, 2009 Sixth International Conference on Information Technology: New Generations.

[31]  C. J. Harwood Test Process Improvement: A Practical Step‐by‐Step Guide to Structured Testing , 2000 .

[32]  K. Eisenhardt Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[33]  Ossi Taipale,et al.  Test case selection and prioritization: risk-based or design-based? , 2010, ESEM '10.