Development of a valid and reliable software customization model for SaaS quality through iterative method: perspectives from academia

Despite the benefits of standardization, the customization of Software as a Service (SaaS) application is also essential because of the many unique requirements of customers. This study, therefore, focuses on the development of a valid and reliable software customization model for SaaS quality that consists of (1) generic software customization types and a list of common practices for each customization type in the SaaS multi-tenant context, and (2) key quality attributes of SaaS applications associated with customization. The study was divided into three phases: the conceptualization of the model, analysis of its validity using SaaS academic-derived expertise, and evaluation of its reliability by submitting it to an internal consistency reliability test conducted by software-engineer researchers. The model was initially devised based on six customization approaches, 46 customization practices, and 13 quality attributes in the SaaS multi-tenant context. Subsequently, its content was validated over two rounds of testing after which one approach and 14 practices were removed and 20 practices were reformulated. The internal consistency reliability study was thereafter conducted by 34 software engineer researchers. All constructs of the content-validated model were found to be reliable in this study. The final version of the model consists of 6 constructs and 44 items. These six constructs and their associated items are as follows: (1) Configuration (eight items), (2) Composition (four items), (3) Extension (six items), 4) Integration (eight items), (5) Modification (five items), and (6) SaaS quality (13 items). The results of the study may contribute to enhancing the capability of empirically analyzing the impact of software customization on SaaS quality by benefiting from all resultant constructs and items.

[1]  Abu Bakar Sultan,et al.  SaaS quality of service attributes , 2014 .

[2]  Hai Jin,et al.  QoS-Driven Service Selection for Multi-tenant SaaS , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[3]  Jos van Hillegersberg,et al.  A pluggable service platform architecture for e-commerce , 2016, Inf. Syst. E Bus. Manag..

[4]  John Grundy,et al.  TOSSMA: A Tenant-Oriented SaaS Security Management Architecture , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[5]  Diane M. Strong,et al.  A framework for evaluating ERP implementation choices , 2004, IEEE Transactions on Engineering Management.

[6]  Vijay Kumar Shrotryia,et al.  Content Validity of Assessment Instrument for Employee Engagement , 2019, SAGE Open.

[7]  Jun Han,et al.  A Feature-Based Framework for Developing and Provisioning Customizable Web Services , 2016, IEEE Transactions on Services Computing.

[8]  Abdul Azim Abd Ghani,et al.  The five Ws taxonomy on customization of software as a service applications , 2018 .

[9]  Gregory J. Skulmoski,et al.  Journal of Information Technology Education the Delphi Method for Graduate Research , 2022 .

[10]  Martin Molina,et al.  A tenant-based resource allocation model for scaling Software-as-a-Service applications over cloud computing infrastructures , 2013, Future Gener. Comput. Syst..

[11]  Du Wan Cheun,et al.  A Quality Model for Evaluating Software-as-a-Service in Cloud Computing , 2009, 2009 Seventh ACIS International Conference on Software Engineering Research, Management and Applications.

[12]  Qingzhong Li,et al.  Process Customization Framework in SaaS Applications , 2013, 2013 10th Web Information System and Application Conference.

[13]  Wouter Joosen,et al.  Feature Models at Run Time: Feature Middleware for Multi-tenant SaaS applications , 2014, Models@run.time.

[14]  Urs Andelfinger,et al.  Applying software product lines to create customizable software-as-a-service applications , 2011, SPLC '11.

[15]  ABDULRAZZAQ QASEM ALI,et al.  EMPIRICAL STUDIES ON THE IMPACT OF SOFTWARE CUSTOMIZATION ON QUALITY ATTRIBUTES: A SYSTEMATIC REVIEW , 2019 .

[16]  Minou Parhizkar.  Impact analysis of enterprise resource planning post-implementation modifications , 2016 .

[17]  Filip De Turck,et al.  Allocating resources for customizable multi-tenant applications in clouds using dynamic feature placement , 2015, Future Gener. Comput. Syst..

[18]  Nor'ashikin Ali,et al.  Toward understanding individuals’ acceptance of internet of things –based services: Developing an instrument to measure the acceptance of smart meters , 2018 .

[19]  Hamid Alavi Majd,et al.  Details of content validity and objectifying it in instrument development , 2014 .

[20]  Gui Lohmann,et al.  Expert validation of a teamwork assessment rubric: A modified Delphi study. , 2016, Nurse education today.

[21]  Muhammad Wasim Bhatti,et al.  Global Monitoring and Control: A Process Improvement Framework for Globally Distributed Software Development Teams , 2017 .

[22]  Elarbi Badidi.  A Framework for Software-as-a-Service Selection and Provisioning , 2013, ArXiv.

[23]  A. Abbaszadeh,et al.  Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. , 2015, Journal of caring sciences.

[24]  Ren Xiaojun,et al.  SaaS Template Evolution Model Based on Tenancy History , 2013, 2013 Third International Conference on Intelligent System Design and Engineering Applications.

[25]  B. J. Pine,et al.  The four faces of mass customization. , 1997, Harvard business review.

[26]  Mervat Abu-Elkheir,et al.  Classifying Requirements for Variability Optimization in Multitenant Applications , 2014, 2014 IEEE 6th International Conference on Cloud Computing Technology and Science.

[27]  Bin Zhang,et al.  Personalized Modeling for SaaS Based on Extended WSCL , 2010, APSCC.

[28]  Mingdi Xin,et al.  Software-as-a-Service Model: Elaborating Client-Side Adoption Factors , 2008, ICIS.

[29]  M. Lynn.  Determination and quantification of content validity. , 1986, Nursing research.

[30]  T. Davenport.  Putting the enterprise into the enterprise system. , 1998, Harvard business review.

[31]  Wei-Tek Tsai,et al.  SaaS Multi-tenant Application Customization , 2013, 2013 IEEE Seventh International Symposium on Service-Oriented System Engineering.

[32]  Arturo Molina,et al.  Application Development over Software-as-a-Service Platforms , 2008, 2008 The Third International Conference on Software Engineering Advances.

[33]  Mika Mäntylä,et al.  A Self-assessment Instrument for Assessing Test Automation Maturity , 2019, EASE.

[34]  António Miguel Rosado da Cruz,et al.  An Architectural Model for Customizing the Business Logic of SaaS Applications , 2013, ICSOFT.

[35]  Wei Sun,et al.  Software as a Service: Configuration and Customization Perspectives , 2008, 2008 IEEE Congress on Services Part II (services-2 2008).

[36]  Wouter Joosen,et al.  Efficient customization of multi-tenant Software-as-a-Service applications with service lines , 2014, J. Syst. Softw..

[37]  Stefan Walraven,et al.  Middleware and Methods for Customizable SaaS , 2014 .

[38]  Wouter Joosen,et al.  Context-oriented programming for customizable SaaS applications , 2012, SAC '12.

[39]  M. Lynne Markus,et al.  Tailoring ERP systems: a spectrum of choices and their implications , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[40]  Filip De Turck,et al.  Developing and managing customizable Software as a Service using feature model conversion , 2012, 2012 IEEE Network Operations and Management Symposium.

[41]  Cheryl Tatano Beck,et al.  The content validity index: are you sure you know what's being reported? Critique and recommendations. , 2006, Research in nursing & health.

[42]  Mona Misfer Al-Shardan,et al.  Configuration as a Service in Multi-Tenant Enterprise Resource Planning System , 2015 .

[43]  Mahmoud Al-Qutayri,et al.  SaaS Dynamic Evolution Based on Model-Driven Software Product Lines , 2014, 2014 IEEE 6th International Conference on Cloud Computing Technology and Science.

[44]  L. Davis.  Instrument review: Getting the most from a panel of experts , 1992 .

[45]  Wouter Joosen,et al.  Middleware for Customizable Multi-staged Dynamic Upgrades of Multi-tenant SaaS Applications , 2015, 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC).

[46]  F. Hasson,et al.  A critical review of the Delphi technique as a research methodology for nursing. , 2001, International journal of nursing studies.

[47]  Yong Shi,et al.  Process customization based on dependent topology in Software as a Service model , 2010, The 2nd International Conference on Software Engineering and Data Mining.

[48]  Yuliang Shi,et al.  Hybrid Fragmentation to Preserve Data Privacy for SaaS , 2011, 2011 Eighth Web Information Systems and Applications Conference.

[49]  Amber J. Fletcher,et al.  Using the Delphi Method for Qualitative, Participatory Action Research in Health Leadership , 2014 .

[50]  Djamal Ziani,et al.  A new framework for customizing ERP systems in a multi tenant SaaS environment , 2015, 2015 2nd World Symposium on Web Applications and Networking (WSWAN).

[51]  Wei-Tek Tsai,et al.  OIC: Ontology-based intelligent customization framework for SaaS , 2010, 2010 IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[52]  Christiane Gresse von Wangenheim,et al.  Discovering Software Process and Product Quality Criteria in Software as a Service , 2010, PROFES.

[53]  Paulo Henrique de Souza Bermejo,et al.  SaaSQuality - A method for Quality Evaluation of Software as a Service (SaaS) , 2013 .

[54]  Zibin Zheng,et al.  Cloud model for service selection , 2011, 2011 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[55]  Shawish Ahmed,et al.  Integrated QoS Utility-Based Model for Cloud Computing Service Provider Selection , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops.

[56]  Marc N. Haines.  Understanding Enterprise System Customization: An Exploration of Implementation Realities and the Key Influence Factors , 2009, Inf. Syst. Manag..

[57]  Souraya Sidani,et al.  Content validity of a process evaluation checklist to measure intervention implementation fidelity of the EPIC intervention. , 2010, Worldviews on evidence-based nursing.

[58]  Srinarayan Sharma,et al.  Impact of customization over software quality in ERP projects: an empirical study , 2016, Software Quality Journal.

[59]  Abdul Azim Abd Ghani,et al.  Customization of Software as a Service Application: Problems and Objectives , 2018, Journal of Computer Science & Computational Mathematics.

[60]  Lars Mathiassen,et al.  Process Mass Customization in a Global Software Firm , 2014, IEEE Software.

[61]  Elizabeth Chang,et al.  Conceptual SLA framework for cloud computing , 2010, 4th IEEE International Conference on Digital Ecosystems and Technologies.

[62]  A. Shafie,et al.  Consensus among healthcare stakeholders on a collaborative medication therapy management model for chronic diseases in Malaysia; A Delphi study , 2019, PloS one.

[63]  Alexander Zeier,et al.  MapperMania: A Framework for Native Multi-Tenancy Business Object Mapping to a Persistent Data Source , 2009, AMCIS.

[64]  Wouter Joosen,et al.  A Middleware Layer for Flexible and Cost-Efficient Multi-tenant Applications , 2011, Middleware.

[65]  Jundian Song,et al.  A QoS Evaluation Model for Test-Bed in the Cloud Computing Environment , 2012, ICEBE.

[66]  Wouter Joosen,et al.  Variability middleware for multi-tenant SaaS applications: a research roadmap for service lines , 2015, SPLC.

[67]  Kamaluddin Hj Talib,et al.  Identifying the relevant features of the National Digital Cadastral Database (NDCDB) for spatial analysis by using the Delphi Technique , 2018 .

[68]  Marijn Janssen,et al.  Design Choices Underlying the Software as a Service (SaaS) Business Model from the User Perspective: Exploring the Fourth Wave of Outsourcing , 2012, J. Univers. Comput. Sci..

[69]  Ralph Mietzner.  Using variability descriptors to describe customizable SaaS application templates , 2008 .

[70]  Uwe Aßmann,et al.  Towards modeling a variable architecture for multi-tenant SaaS-applications , 2012, VaMoS.

[71]  Srinarayan Sharma,et al.  Efficiency analysis of ERP packages - A customization perspective , 2016, Comput. Ind..

[72]  Annamaria Bagnasco,et al.  Developing an instrument to measure emotional behaviour abilities of meaningful learning through the Delphi technique , 2017, Journal of advanced nursing.

[73]  Rory V. O'Connor,et al.  An examination of personality traits and how they impact on software development teams , 2017, Inf. Softw. Technol..

[74]  Ashraf A. Shahin.  Multi-Dimensional Customization Modelling Based On Metagraph For Saas Multi-Tenant Applications , 2014, ArXiv.

[75]  Farookh Khadeer Hussain,et al.  An integrated personalization framework for SaaS-based cloud services , 2015, Future Gener. Comput. Syst..

[76]  Gyoo Gun Lim,et al.  Using balanced scorecards for the evaluation of "Software-as-a-service" , 2013, Inf. Manag..

[77]  Jun Han,et al.  Software-Defined Service Networking: Runtime Sharing with Performance Differentiation in Multi-tenant SaaS Applications , 2015, 2015 IEEE International Conference on Services Computing.

[78]  Atieh Khanjani.  Quality of service model for software as a service in cloud computing from users' and providers' perspectives , 2015 .

[79]  Wonjae Lee,et al.  A Multi-tenant Web Application Framework for SaaS , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[80]  Holger Wache,et al.  Capturing Customers' Requirements towards Mixed-Tenancy Deployments of SaaS-Applications , 2013, 2013 IEEE Sixth International Conference on Cloud Computing.

[81]  C. Wynd,et al.  Two Quantitative Approaches for Estimating Content Validity , 2003, Western journal of nursing research.

[82]  Beijun Shen,et al.  Mass data processing and personalized services in Shanghai e-commerce credit evaluation platform , 2014, 2014 IEEE International Conference on Progress in Informatics and Computing.

[83]  R. Rajmohan,et al.  QoS evaluation for web services in cloud computing , 2012, 2012 Third International Conference on Computing, Communication and Networking Technologies (ICCCNT'12).

[84]  Yong Zhang,et al.  Towards high level SaaS maturity model: Methods and case study , 2009, 2009 IEEE Asia-Pacific Services Computing Conference (APSCC).

[85]  Tianyi Zang,et al.  Variable service process by feature meta-model for SaaS application , 2012, 2012 International Conference on Green and Ubiquitous Technology.

[86]  C. Harris,et al.  Development of an evidence‐based protocol for care of pilonidal sinus wounds healing by secondary intent using a modified reactive Delphi procedure. Part one: the literature review * , 2012, International wound journal.

[87]  J. Landeta.  Current validity of the Delphi method in social sciences , 2006 .

[88]  Pierre Pluye,et al.  Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. , 2019, Journal of clinical epidemiology.

[89]  Frank Leymann,et al.  EAI as a Service - Combining the Power of Executable EAI Patterns and SaaS , 2008, 2008 12th International IEEE Enterprise Distributed Object Computing Conference.

[90]  François Lustman,et al.  A change impact model for changeability assessment in object-oriented software systems , 2002, Sci. Comput. Program..

[91]  Wei-Tek Tsai,et al.  EasySaaS: A SaaS development framework , 2011, 2011 IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[92]  Slinger Jansen,et al.  Variability in Multi-tenant Environments: Architectural Design Patterns from Industry , 2011, ER Workshops.

[93]  Nadir K. Salih,et al.  Modeling and Self-Configuring SaaS Application , 2016, ArXiv.

[94]  Wouter Joosen,et al.  Scalable and manageable customization of workflows in multi-tenant SaaS offerings , 2016, SAC.

[95]  Min Xu,et al.  APE : An Automated Performance Engineering Process for Software as a Service Environments , .

[96]  Alexander Zeier,et al.  Customizing Enterprise Software as a Service Applications: Back-End Extension in a Multi-tenancy Environment , 2009, ICEIS.

[97]  Wenyu Liu,et al.  New model of SaaS: SaaS with tenancy agency , 2010, 2010 2nd International Conference on Advanced Computer Control.

[98]  Alfons Kemper,et al.  Extensibility and Data Sharing in evolving multi-tenant databases , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[99]  L. Cohen,et al.  Research Methods in Education , 1980 .

[100]  Joseph Barjis,et al.  Extending multi-tenant architectures: a database model for a multi-target support in SaaS applications , 2016, Enterp. Inf. Syst..

[101]  Frank Leymann,et al.  Generation of BPEL Customization Processes for SaaS Applications from Variability Descriptors , 2008, 2008 IEEE International Conference on Services Computing.

[102]  Kuo Zhang,et al.  A Policy-Driven Approach for Software-as-Services Customization , 2007, The 9th IEEE International Conference on E-Commerce Technology and The 4th IEEE International Conference on Enterprise Computing, E-Commerce and E-Services (CEC-EEE 2007).

[103]  Ashraf Zia,et al.  Identifying Key Challenges in Performance Issues in Cloud Computing , 2012 .

[104]  Yuliang Shi,et al.  A Multi-tenant Oriented Business Process Customization System , 2009, 2009 International Conference on New Trends in Information and Service Science.

[105]  Soo Dong Kim,et al.  A Systematic Process for Developing High Quality SaaS Cloud Services , 2009, CloudCom.

[106]  Judith A Effken,et al.  Validating the personal health records adoption model using a modified e-Delphi. , 2013, Journal of advanced nursing.

[107]  Wei-Tek Tsai,et al.  Tenant-centric Sub-Tenancy Architecture in Software-as-a-Service , 2016, CAAI Trans. Intell. Technol..

[108]  Ijcsis.  Reusability Quality Attributes and Metrics of SaaS from Perspective of Business and Provider , 2016 .

[109]  Frank Leymann,et al.  Defining Composite Configurable SaaS Application Packages Using SCA, Variability Descriptors and Multi-tenancy Patterns , 2008, 2008 Third International Conference on Internet and Web Applications and Services.

[110]  André Vasconcelos,et al.  SaaS (Software as a Service) - Infrastructures and Applications in Real Scenarios , 2010 .

[111]  Jun Han,et al.  Sharing with a Difference: Realizing Service-Based SaaS Applications with Runtime Sharing and Variation in Dynamic Software Product Lines , 2013, 2013 IEEE International Conference on Services Computing.

[112]  Zhongbo Jiang,et al.  Study of Software as a Service Support Platform for Small and Medium Businesses , 2011 .

[113]  Qingzhong Li,et al.  A Novel Model Supporting Customization Sharing in SaaS Applications , 2010, 2010 International Conference on Multimedia Information Networking and Security.

[114]  M. S. Kavitha,et al.  Software as a Service in Cloud Computing , 2020 .

[115]  Yuliang Shi,et al.  A Multi-granularity Customization Relationship Model for SaaS , 2009, 2009 International Conference on Web Information Systems and Mining.

[116]  Kuo Zhang,et al.  Software as a Service: An Integration Perspective , 2007, ICSOC.

[117]  Anne Sunikka,et al.  What, Who and Where: Insights into Personalization , 2008, Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008).

[118]  Joseph Beyene,et al.  Identification of pain indicators for infants at risk for neurological impairment: A Delphi consensus study , 2006, BMC pediatrics.

[119]  Hazura Zulzalil,et al.  A Systematic Mapping Study on the Customization Solutions of Software as a Service Applications , 2019, IEEE Access.

[120]  Filip De Turck,et al.  Feature-based application development and management of multi-tenant applications in clouds , 2014, SPLC.