Aiming at methods' wider adoption: Applicability determinants and metrics

Abstract Numerous computer science methods and techniques have been proposed by the scientific community. However, depending on the domain, only their minor fraction has met wider adoption. This paper brings attention to the concept of applicability — the notion that is well acknowledged in the scientific field but have not been analysed with respect to determinants, metrics and systematisation. The primary objective of the study was to identify applicability determinants and metrics and consolidate them into a taxonomy, based on a systematic process. To achieve the objective, a methodological literature review supported with operationalisation activities were applied. As a result, more than thirty descriptors of applicability were introduced in the proposed applicability taxonomy, and a similar number of metrics has been elicited. Based on them, a questionnaire for evaluation of the method’s applicability was created and applied to the evaluation of a currently developed cybersecurity risk assessment method that aims at broader market uptake. The analysis provided indications on its further developments, but also findings of a more universal character.

[1]  Ina Schieferdecker,et al.  A taxonomy of risk-based testing , 2014, International Journal on Software Tools for Technology Transfer.

[2]  Frank Werner,et al.  Risk Assessment and Security Testing of Large Scale Networked Systems with RACOMAT , 2015, RISK.

[3]  Jürgen Großmann,et al.  Combining Security Risk Assessment and Security Testing Based on Standards , 2015, RISK.

[4]  Pedro Luiz Pizzigatti Corrêa,et al.  Considerations about the efficiency and sufficiency of the utilization of the Scrum methodology: A survey for analyzing results for development teams , 2021, Comput. Sci. Rev..

[5]  Rex B. Kline,et al.  Usability measurement and metrics: A consolidated model , 2006, Software Quality Journal.

[6]  Katharina Barkley.  Does one size fit all? The applicability of situational crisis communication theory in the Japanese context , 2020 .

[7]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[8]  Nicholas J. Car,et al.  USING decision models to enable better irrigation Decision Support Systems , 2018, Comput. Electron. Agric..

[9]  Richard Lippmann,et al.  Agent-based simulation for assessing network security risk due to unauthorized hardware , 2015, SpringSim.

[10]  Carol M. Barnum.  Preparing for usability testing , 2011 .

[11]  Jan H. P. Eloff,et al.  A comparative framework for risk analysis methods , 1993, Comput. Secur..

[12]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research , 2007 .

[13]  Jin B. Hong,et al.  A survey on the usability and practical applications of Graphical Security Models , 2017, Comput. Sci. Rev..

[14]  Kevin Jones,et al.  A review of cyber security risk assessment methods for SCADA systems , 2016, Comput. Secur..

[15]  Eric Walling,et al.  Developing successful environmental decision support systems: Challenges and best practices. , 2020, Journal of environmental management.

[16]  Joint Task Force Transformation Initiative.  Guide for Conducting Risk Assessments , 2012 .

[17]  L. Morales,et al.  Applicability of LAMDA as classification model in the oil production , 2019, Artificial Intelligence Review.

[18]  Pieter H. Hartel,et al.  Current established risk assessment methodologies and tools , 2013 .

[19]  Eleni Berki,et al.  Usability, security and trust in password managers: A quest for user-centric properties and features , 2019, Comput. Sci. Rev..

[20]  Bin Tang,et al.  Rateless802.11: Extending WiFi applicability in extremely poor channels , 2020, Comput. Networks.

[21]  Alan G. Downe,et al.  Determinants of computer usage among educators: A comparison between the UTAUT and TAM models , 2011, 2011 National Postgraduate Conference.

[22]  C. D. Jaidhar,et al.  Applicability of machine learning in spam and phishing email filtering: review and approaches , 2020, Artificial Intelligence Review.

[23]  Jason Edwin Stamp,et al.  A classification scheme for risk assessment methods. , 2004 .

[24]  James R. Lewis,et al.  The System Usability Scale: Past, Present, and Future , 2018, Int. J. Hum. Comput. Interact..

[25]  Jurek Kirakowski,et al.  Measuring the Usability of Web Sites , 1998 .

[26]  Les Labuschagne,et al.  A framework for comparing different information security risk analysis methodologies , 2005 .

[27]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[28]  S. Lloyd,et al.  Measures of complexity: a nonexhaustive list , 2001 .

[29]  Einar Snekkenes,et al.  A framework for estimating information security risk assessment method completeness , 2017, International Journal of Information Security.

[30]  Jeff Sauro,et al.  Standardized Usability Questionnaires , 2012 .

[31]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[32]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[33]  Aleksandr Mylläri,et al.  On the Applicability of Pairwise Separations Method in Astronomy: Influence of the Noise in Data , 2019, Math. Comput. Sci..

[34]  Ahmed Patel,et al.  A review of security assessment methodologies in industrial control systems , 2019, Inf. Comput. Secur..

[35]  Geoffrey Hughes,et al.  Longman Dictionary of Contemporary English , 2013 .

[36]  Pearl Brereton,et al.  A systematic review of systematic review process research in software engineering , 2013, Inf. Softw. Technol..

[37]  Jan Tyrychtr,et al.  The current state of the issue of information needs and dispositions among small Czech farms , 2017 .

[38]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[39]  Shingo Asamoto,et al.  Applicability of machine learning to a crack model in concrete bridges , 2020, Comput. Aided Civ. Infrastructure Eng..

[40]  R. G. N. Meegama,et al.  Applicability of a Single Depth Sensor in Real-Time 3D Clothes Simulation: Augmented Reality Virtual Dressing Room Using Kinect Sensor , 2020, Adv. Hum. Comput. Interact..

[41]  Alexey L. Lastovetsky,et al.  A Survey of Communication Performance Models for High-Performance Computing , 2019, ACM Comput. Surv..

[42]  S. Johansson,et al.  Digital production innovation projects – The applicability of managerial controls under high levels of complexity and uncertainty , 2020 .

[43]  Martin Ibl,et al.  Complexity analysis of business processes , 2017, 2017 International Conference on Information and Digital Technologies (IDT).

[44]  Harris Papadopoulos,et al.  Measures of Complexity , 2015, Springer International Publishing.

[45]  Kai Virtanen,et al.  On Theoretical Principle and Practical Applicability of Ranked Nodes Method for Constructing Conditional Probability Tables of Bayesian Networks , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[46]  Luiza Fabisiak,et al.  Comparative Analysis of Information Security Assessment and Management Methods , 2012 .

[47]  Vivek Agrawal,et al.  A Comparative Study on Information Security Risk Analysis Methods , 2017, J. Comput..

[48]  Antonio Ruiz-Martínez,et al.  Systematic literature review on the state of the art and future research work in anonymous communications systems , 2017, Comput. Electr. Eng..

[49]  Kyungho Lee,et al.  An advanced approach to security measurement system , 2015, The Journal of Supercomputing.

[50]  Vitor Nazário Coelho,et al.  Multi-agent systems applied for energy systems integration: State-of-the-art applications and trends in microgrids , 2017 .

[51]  Burkhard Hoppenstedt,et al.  Applicability of Immersive Analytics in Mixed Reality: Usability Study , 2019, IEEE Access.

[52]  W Wim Zeiler,et al.  On the application of multi-agent systems in buildings for improved building operations, performance and smart grid interaction: a survey , 2015 .

[53]  Ioannis K. Argyros,et al.  Extending the applicability of Newton's method for a class of boundary value problems using the shooting method , 2020, Appl. Math. Comput..

[54]  Jorge S. Cardoso,et al.  Business Process Control-Flow Complexity: Metric, Evaluation, and Validation , 2008, Int. J. Web Serv. Res..

[55]  Marian Makins,et al.  Collins English dictionary , 1991 .

[56]  Steen Nielsen.  Reflections on the applicability of business analytics for management accounting – and future perspectives for the accountant , 2018, Journal of Accounting & Organizational Change.

[57]  James R. Lewis,et al.  Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ , 1991, SGCH.

[58]  Claudia Eckert,et al.  Attack Graph-Based Assessment of Exploitability Risks in Automotive On-Board Networks , 2018, ARES.

[59]  Dimitris Gritzalis,et al.  Exiting the Risk Assessment Maze , 2018, ACM Comput. Surv..

[60]  Giannopoulos Georgios,et al.  Risk assessment methodologies for Critical Infrastructure Protection. Part I: A state of the art , 2012 .

[61]  Latifa Ben Arfa Rabai,et al.  A Survey of Quantitative Security Risk Analysis Models for Computer Systems , 2018, ICAAI 2018.

[62]  Victoria Beltran,et al.  Decision support systems for agriculture 4.0: Survey and challenges , 2020, Comput. Electron. Agric..

[63]  Ivan Martinovic,et al.  On the Applicability of Satellite-Based Air Traffic Control Communication for Security , 2019, IEEE Communications Magazine.

[64]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[65]  E. G. Domingues,et al.  Calculation of system complexity based on the connections: Methodology and applications , 2016, 2016 IEEE 16th International Conference on Environment and Electrical Engineering (EEEIC).

[66]  Stephen L. Dorton,et al.  Effective Variety? for whom (or what)? A folk theory on interface complexity and situation awareness , 2017, 2017 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA).

[67]  Terje Aven,et al.  The risk concept - historical and recent development trends , 2012, Reliab. Eng. Syst. Saf..

[68]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[69]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[70]  Yevgeniya Kovalchuk,et al.  Machine learning and multi-agent systems in oil and gas industry applications: A survey , 2019, Comput. Sci. Rev..

[71]  Miquel Sànchez-Marrè,et al.  Decision support systems (DSS) for wastewater treatment plants - A review of the state of the art. , 2019, Bioresource technology.

[72]  Kurt Sandkuhl,et al.  An Analysis of Applicability using Quality Metrics for Ontologies on Ontology Design Patterns , 2015, Intell. Syst. Account. Finance Manag..

[73]  Václav Kotlan,et al.  Applicability and comparison of surrogate techniques for modeling of selected heating problems , 2019, Comput. Math. Appl..