Adaptive Fault Detection for Testing Tenant Applications in Multi-tenancy SaaS Systems

SaaS (Software-as-a-Service) often uses multi-tenancy architecture (MTA) where tenant developers compose their applications online using the components stored in the SaaS database. Tenant applications need to be tested, and combinatorial testing can be used. While numerous combinatorial testing techniques are available, most of them produce static sequences of test configurations and their goal is often to provide sufficient coverage such as 2-way interaction coverage. But the goal of SaaS testing is to identify those compositions that are faulty for tenant applications. This paper proposes an adaptive test configuration generation algorithm AR (Adaptive Reasoning) that can rapidly identify those faulty combinations so that those faulty combinations cannot be selected by tenant developers for composition. The AR algorithm has been evaluated by both simulation and real experimentation using a MTA SaaS sample running on GAE (Google App Engine). Both the simulation and experiment showed show that the AR algorithm can identify those faulty combinations rapidly. Whenever a new component is submitted to the SaaS database, the AR algorithm can be applied so that any faulty interactions with new components can be identified to continue to support future tenant applications.

[1]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[2]  Frank Leymann,et al.  Generation of BPEL Customization Processes for SaaS Applications from Variability Descriptors , 2008, 2008 IEEE International Conference on Services Computing.

[3]  Qian Huang,et al.  Ontology-based Dynamic Process Collaboration in Service-Oriented Architecture , 2007, IEEE International Conference on Service-Oriented Computing and Applications (SOCA '07).

[4]  Wei-Tek Tsai,et al.  Scalable Architectures for SaaS , 2012, 2012 IEEE 15th International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing Workshops.

[5]  Wei-Tek Tsai,et al.  OIC: Ontology-based intelligent customization framework for SaaS , 2010, 2010 IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[6]  Lucia Moura,et al.  Locating Errors Using ELAs, Covering Arrays, and Adaptive Testing Algorithms , 2009, SIAM J. Discret. Math..

[7]  Information Security, Coding Theory and Related Combinatorics , 2011, Information Security, Coding Theory and Related Combinatorics.

[8]  Charles J. Colbourn,et al.  Locating and detecting arrays for interaction faults , 2008, J. Comb. Optim..

[9]  Yao Zhang,et al.  Semantic-Based Test Oracles , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference.

[10]  Charles J. Colbourn,et al.  Covering arrays and hash families , 2011, Information Security, Coding Theory and Related Combinatorics.

[11]  Douglas C. Schmidt,et al.  Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance , 2007, IEEE Transactions on Software Engineering.

[12]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[13]  Yashwant K. Malaiya,et al.  Antirandom testing: getting the most out of black-box testing , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[14]  Wei-Tek Tsai,et al.  SaaS performance and scalability evaluation in clouds , 2011, Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE).

[15]  Wei-Tek Tsai,et al.  Towards a scalable and robust multi-tenancy SaaS , 2010, Internetware.

[16]  Wei-Tek Tsai,et al.  A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation , 2012, 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings.

[17]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[18]  Myra B. Cohen,et al.  Towards incremental adaptive covering arrays , 2007, ESEC-FSE companion '07.

[19]  Victor V. Kuliamin,et al.  A survey of methods for constructing covering arrays , 2011, Programming and Computer Software.

[20]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[21]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[22]  Torsten Grust,et al.  Multi-tenant databases for software as a service: schema-mapping techniques , 2008, SIGMOD Conference.

[23]  Neil Genzlinger.  A. and Q , 2006 .