Mind the gaps: increasing the impact of IS research on ISD performance improvement

Poor performance has pervaded the last forty years of software development, evident across industry sectors, project size, budget, geographic location, system quality and functionality, and exacerbated by increased criticality of IT in organizational mission and strategy. A significant body of research has investigated the potential of emerging development methodologies to address these shortcomings but the effectiveness of these methods is largely supported by anecdotal evidence. At the same time, metrics and measurement are known to affect ISD performance but the existing literature on ISD metrics is misaligned with practitioners’ needs, leading to a lack of clarity about ISD metrics in practice. This paper presents an interdisciplinary literature review on ISD metrics to identify the underlying reasons for this misalignment and evaluate the extent to which existing literature can be used to better understand the impact of emerging software development methodologies on ISD performance.

[1]  Warren Harrison A flexible method for maintaining software metrics data: a universal metrics repository , 2004, J. Syst. Softw..

[2]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[3]  Janice Singer,et al.  Hipikat: a project memory for software development , 2005, IEEE Transactions on Software Engineering.

[4]  Xiaotong Li,et al.  Software development risk and project performance measurement: Evidence in Korea , 2007, J. Syst. Softw..

[5]  Michael R. Clarkson,et al.  Formal Methods Application: An Empirical Tale of Software Development , 2002, IEEE Trans. Software Eng..

[6]  Kieran Conboy,et al.  Agility from First Principles: Reconstructing the Concept of Agility in Information Systems Development , 2009, Inf. Syst. Res..

[7]  Walker Royce CMM vs . CMMI : From Conventional to Modern Software Management by , 2002 .

[8]  Emilia Mendes,et al.  Investigating Web size metrics for early Web cost estimation , 2005, J. Syst. Softw..

[9]  Laurie A. Williams,et al.  Validating software metrics: A spectrum of philosophies , 2012, TSEM.

[10]  Mohammad Alshayeb,et al.  An Empirical Validation of Object-Oriented Metrics in Two Different Iterative Software Processes , 2003, IEEE Trans. Software Eng..

[11]  Oisín Cawley,et al.  "Leagile" software development: An experience report analysis of the application of lean approaches in agile software development , 2012, J. Syst. Softw..

[12]  Reiner R. Dumke,et al.  Software Metrics for Agile Software Development , 2008, 19th Australian Conference on Software Engineering (aswec 2008).

[13]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[14]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[15]  Deborah J. Armstrong,et al.  Person-Job Cognitive Style Fit for Software Developers: The Effect on Strain and Performance , 2005, J. Manag. Inf. Syst..

[16]  Miroslaw Staron,et al.  Measuring the impact of changes to the complexity and coupling properties of automotive software systems , 2013, J. Syst. Softw..

[17]  M. Lynne Markus,et al.  If we build it, they will come: Designing information systems that people want to use , 1994 .

[18]  Kai Petersen,et al.  A decision support framework for metrics selection in goal-based measurement programs: GQM-DSFMS , 2013, J. Syst. Softw..

[19]  Ye Yang,et al.  Quantitative defects management in iterative development with BiDefect , 2009, Softw. Process. Improv. Pract..

[20]  Miroslaw Staron,et al.  Critical role of measures in decision processes: Managerial and technical measures in the context of large software development organizations , 2012, Inf. Softw. Technol..

[21]  Avinash C. Kak,et al.  Metrics for Measuring the Quality of Modularization of Large-Scale Object-Oriented Software , 2008, IEEE Transactions on Software Engineering.

[22]  Chin-Tsai Lin,et al.  Model for measuring quality of software in DVRS using the gap concept and fuzzy schemes with GA , 2006, Inf. Softw. Technol..

[23]  Michele Marchesi,et al.  Study of the Evolution of an Agile Project Featuring a Web Application Using Software Metrics , 2008, PROFES.

[24]  Weidong Xia,et al.  Complexity of Information Systems Development Projects: Conceptualization and Measurement Development , 2005, J. Manag. Inf. Syst..

[25]  Marko Palviainen,et al.  The reliability estimation, prediction and measuring of component-based software , 2011, J. Syst. Softw..

[26]  Michael Lang,et al.  Complexity in Information Systems Development , 2017 .

[27]  Brian Fitzgerald,et al.  Method and developer characteristics for effective agile method tailoring: A study of XP expert opinion , 2010, TSEM.

[28]  V. Basili Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[29]  Weidong Xia,et al.  The ability of information systems development project teams to respond to business and technology changes: a study of flexibility measures , 2005, Eur. J. Inf. Syst..

[30]  Steve Sawyer,et al.  Social interactions of information systems development teams: a performance perspective , 2010, Inf. Syst. J..

[31]  Yves Pigneur,et al.  Improving globally distributed software development and support processes – A workflow view , 2013, J. Softw. Evol. Process..

[32]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[33]  Chris Barry,et al.  Finding a home for web-based information systems - perusing the landscape , 2003, ECIS.

[34]  Tong-Seng Quah,et al.  Prediction of software development faults in PL/SQL files using neural network models , 2004, Inf. Softw. Technol..

[35]  Antonio Vallecillo,et al.  Measuring the usability of software components , 2006, J. Syst. Softw..

[36]  Thanwadee Sunetnanta,et al.  Quality Indicators on Global Software Development Projects: Does "Getting to Know You" Really Matter? , 2009, 2009 Fourth IEEE International Conference on Global Software Engineering.

[37]  Ioannis Stamelos,et al.  Layer assessment of object-oriented software: A metric facilitating white-box reuse , 2013, J. Syst. Softw..

[38]  Richard Lai,et al.  Component Point: A system-level size measure for Component-Based Software Systems , 2010, J. Syst. Softw..

[39]  Mark Keil,et al.  Effects of information technology failures on the market value of firms , 2009, J. Strateg. Inf. Syst..

[40]  Irit Hadar,et al.  Why and how can human-related measures support software development processes? , 2008, J. Syst. Softw..

[41]  Lars Mathiassen,et al.  Cultivation and engineering of a software metrics program , 2003, Inf. Syst. J..

[42]  Per Runeson,et al.  A spiral process model for case studies on software quality monitoring - method and metrics , 2007, Softw. Process. Improv. Pract..

[43]  C. van Koten,et al.  Bayesian statistical effort prediction models for data-centred 4GL software development , 2006, Inf. Softw. Technol..

[44]  Jeannette M. Wing,et al.  An Attack Surface Metric , 2011, IEEE Transactions on Software Engineering.

[45]  Lars Lundberg,et al.  Software development productivity on a new platform: an industrial case study , 2005, Inf. Softw. Technol..

[46]  Ioannis Stamelos,et al.  BBN based approach for improving the software development process of an SME - a case study , 2010, J. Softw. Maintenance Res. Pract..

[47]  Jeff Sutherland,et al.  Scrum Metrics for Hyperproductive Teams: How They Fly like Fighter Aircraft , 2013, 2013 46th Hawaii International Conference on System Sciences.

[48]  Letha H. Etzkorn,et al.  A comparison of cohesion metrics for object-oriented systems , 2004, Inf. Softw. Technol..

[49]  Serge Demeyer,et al.  On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test , 2007, IEEE Trans. Software Eng..

[50]  Tore Dybå,et al.  Applying Systematic Reviews to Diverse Study Types: An Experience Report , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[51]  Qin Zhang,et al.  Improving software development management through software project telemetry , 2005, IEEE Software.

[52]  Venugopal Balijepally,et al.  Theoretical reflections on agile development methodologies , 2007, Commun. ACM.

[53]  Witold Pedrycz,et al.  Practical assessment of the models for identification of defect-prone classes in object-oriented commercial systems using design metrics , 2003, J. Syst. Softw..

[54]  Ming Li,et al.  A Ranking of Software Engineering Measures Based on Expert Opinion , 2003, IEEE Trans. Software Eng..

[55]  Jouni Markkula,et al.  How Does a Measurement Programme Evolve in Software Organizations? , 2008, PROFES.

[56]  A. Fink Conducting research literature reviews : from paper to the Internet , 1998 .

[57]  Stephanie D. Teasley,et al.  Rapid Software Development through Team Collocation , 2002, IEEE Trans. Software Eng..

[58]  Deborah Hartmann,et al.  Appropriate agile measurement: using metrics and diagnostics to deliver business value , 2006, AGILE 2006 (AGILE'06).

[59]  Saonee Sarker,et al.  Assessing the relative contribution of the facets of agility to distributed systems development success: an Analytic Hierarchy Process approach , 2009, Eur. J. Inf. Syst..

[60]  Weidong Xia,et al.  Toward Agile: An Integrated Analysis of Quantitative and Qualitative Field Data , 2010, MIS Q..

[61]  Eduardo Miranda,et al.  Agile monitoring using the line of balance , 2010, J. Syst. Softw..

[62]  Chin-Yu Huang,et al.  Enhancing and measuring the predictive capabilities of testing-effort dependent software reliability models , 2008, J. Syst. Softw..

[63]  Lech Madeyski,et al.  The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment , 2010, Inf. Softw. Technol..

[64]  Mayuram S. Krishnan,et al.  Measurement Programs in Software Development: Determinants of Success , 2002, IEEE Trans. Software Eng..

[65]  George Michaelides,et al.  Successful extreme programming: Fidelity to the methodology or good teamworking? , 2013, Inf. Softw. Technol..

[66]  Claes Wohlin,et al.  Global software engineering and agile practices: a systematic review , 2012, J. Softw. Evol. Process..

[67]  Miroslaw Staron,et al.  A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation , 2010, Inf. Softw. Technol..

[68]  Ioannis Stamelos,et al.  Code quality analysis in open source software development , 2002, Inf. Syst. J..

[69]  Hanna Kulas,et al.  Product metrics in agile software development , 2012 .

[70]  Vadlamani Ravi,et al.  Software development cost estimation using wavelet neural networks , 2008, J. Syst. Softw..

[71]  Letha H. Etzkorn,et al.  Empirical Validation of Three Software Metrics Suites to Predict Fault-Proneness of Object-Oriented Classes Developed Using Highly Iterative or Agile Software Development Processes , 2007, IEEE Transactions on Software Engineering.

[72]  Mohammad Alshayeb,et al.  An empirical study of system design instability metric and design evolution in an agile software process , 2005, J. Syst. Softw..

[73]  Pekka Abrahamsson,et al.  ‘Lots done, more to do’: the current state of agile systems development research , 2009, Eur. J. Inf. Syst..

[74]  David W. Binkley,et al.  Increasing diversity: Natural language measures for software fault prediction , 2009, J. Syst. Softw..

[75]  Tung Bui,et al.  Simulating mixed agile and plan-based requirements prioritization strategies: proof-of-concept and practical implications , 2009, Eur. J. Inf. Syst..

[76]  Eduardo Figueiredo,et al.  On the relationship of concern metrics and requirements maintainability , 2012, Inf. Softw. Technol..

[77]  Peter Henderson,et al.  A tool for evaluation of the software development process , 2001, J. Syst. Softw..

[78]  Kai Petersen,et al.  Measuring and predicting software productivity: A systematic map and review , 2011, Inf. Softw. Technol..

[79]  Doo-Hwan Bae,et al.  Measuring behavioral dependency for improving change-proneness prediction in UML-based design models , 2010, J. Syst. Softw..

[80]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[81]  Claes Wohlin,et al.  The effect of moving from a plan-driven to an incremental software development approach with agile practices , 2010, Empirical Software Engineering.

[82]  Gabriele Bavota,et al.  Identifying Extract Class refactoring opportunities using structural and semantic cohesion measures , 2011, J. Syst. Softw..

[83]  Anirudha N. Joshi,et al.  Measuring effectiveness of HCI integration in software development processes , 2010, J. Syst. Softw..

[84]  Norman E. Fenton,et al.  Measurement : A Necessary Scientific Basis , 2004 .

[85]  David Raffo,et al.  Software project management using PROMPT: A hybrid metrics, modeling and utility framework , 2005, Inf. Softw. Technol..

[86]  June M. Verner,et al.  Toward predicting software development success from the perspective of practitioners: an exploratory Bayesian model , 2005, J. Inf. Technol..