On tackling quality threats for the assessment of measurement programs: A case study on the distribution of metric usage and knowledge

Context Measurement programs are subject to changing requirements, causing the need for a robustness to compensate these changes. Existing methods enable the assessment of measurement programs and their robustness with the help of company representatives. Goal: However, it is not clear how much the spread of knowledge among roles and differences between company internal organizations influence the result and quality of the assessment. Thus, it is not known today who and how many representatives should be interviewed to get reliable results. Method: To address these points we performed a case study on the assessment method MeSRAM, spanning 18 interviews with 14 representatives of 4 different organizations within a large international communication company. Results: Managers are able to answer more questions and their answers are more often correct than those of engineers. Furthermore, most errors occur for questions about the use of concrete metrics, while for aspects about the metrics' infrastructure no conflicting answers occurred. We also identified 6 reasons for gaining incorrect answers. Conclusion: The assessment of measurement programs requires additional care to ensure the quality of the results. (C) 2016 Elsevier B.V. All rights reserved.

[1]  Davi Viana,et al.  The influence of human aspects on software process improvement: Qualitative research findings and comparison to previous studies , 2012, EASE.

[2]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[3]  Miroslaw Staron,et al.  Measuring and Visualizing Code Stability -- A Case Study at Three Companies , 2013, 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement.

[4]  Tapani Kilpi,et al.  Implementing a Software Metrics Program at Nokia , 2001, IEEE Softw..

[5]  Kai Petersen,et al.  Lean Software Development Measures and Indicators - A Systematic Mapping Study , 2013, LESS.

[6]  Günther Ruhe,et al.  RELREA - An Analytical Approach for Evaluating Release Readiness , 2014, SEKE.

[7]  Ivan Porres,et al.  A Metrics Model to Measure the Impact of an Agile Transformation in Large Software Development Organizations , 2013, XP.

[8]  Alain Abran,et al.  LIME: A three-dimensional measurement model for life cycle project management , 1999 .

[9]  Peter Schuh.  Integrating Agile Development In The Real World , 2004 .

[10]  Miroslaw Staron,et al.  MeSRAM - A method for assessing robustness of measurement programs in large software development organizations and its industrial evaluation , 2016, J. Syst. Softw..

[11]  Dimitris Christodoulakis,et al.  Measuring perceived software quality , 1997, Inf. Softw. Technol..

[12]  Chia-Ling Lee,et al.  Organization structure, competition and performance measurement systems and their joint effects on performance , 2011 .

[13]  Orit Hazzan,et al.  Agile metrics at the Israeli Air Force , 2005, Agile Development Conference (ADC'05).

[14]  Petri Kettunen,et al.  Adopting key lessons from agile manufacturing to agile software product development—A comparative study , 2009 .

[15]  Félix García,et al.  MIS-PyME software measurement capability maturity model - Supporting the definition of software measurement programs and capability determination , 2010, Adv. Eng. Softw..

[16]  Robert W. Peach.  The ISO 9000 handbook , 1995 .

[17]  Lionel C. Briand,et al.  Exploring the relationships between design measures and software quality in object-oriented systems , 2000, J. Syst. Softw..

[18]  Austen Rainer,et al.  Software Process Improvement Problems in Twelve Software Companies: An Empirical Analysis , 2003, Empirical Software Engineering.

[19]  Lucas Layman,et al.  Motivations and measurements in an agile case study , 2006, J. Syst. Archit..

[20]  Alan R. Hevner,et al.  The impacts of quality and productivity perceptions on the use of software process improvement innovations , 2005, Inf. Softw. Technol..

[21]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[22]  Ivan Porres,et al.  Metrics Functions for Kanban Guards , 2010, 2010 17th IEEE International Conference and Workshops on Engineering of Computer Based Systems.

[23]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[24]  Miroslaw Staron,et al.  Release Readiness Indicator for Mature Agile and Lean Software Development Projects , 2012, XP.

[25]  Félix García,et al.  Implementing a software measurement program in small and medium enterprises: a suitable framework , 2008, IET Softw..

[26]  Jouni Markkula,et al.  How Does a Measurement Programme Evolve in Software Organizations? , 2008, PROFES.

[27]  Tracy Hall,et al.  Practitioner roles in software process improvement: an analysis using grid technique , 2002, Softw. Process. Improv. Pract..

[28]  Gregory G. Dess,et al.  Voluntary Turnover, Social Capital, and Organizational Performance , 2001 .

[29]  Torgeir Dingsøyr,et al.  Managing Knowledge in Global Software Development Projects , 2014, IT Professional.

[30]  Deborah Nightingale,et al.  Toward a better comprehension of Lean metrics for research and product development management , 2014 .

[31]  Alain Abran,et al.  LIME: a three-dimensional software performance measurement model for project management , 2000 .

[32]  M. Lindvall,et al.  Knowledge management in software engineering , 2002, IEEE Software.

[33]  Tracy Hall,et al.  De-motivators for software process improvement: an analysis of practitioners' views , 2003, J. Syst. Softw..

[34]  Kai Petersen.  A Palette of Lean Indicators to Detect Waste in Software Maintenance: A Case Study , 2012, XP.

[35]  Hans van Vliet,et al.  Measurement program success factors revisited , 2001, Inf. Softw. Technol..

[36]  Andrew Glover,et al.  Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series) , 2007 .

[37]  Deborah Hartmann,et al.  Appropriate agile measurement: using metrics and diagnostics to deliver business value , 2006, AGILE 2006 (AGILE'06).

[38]  Alain Abran,et al.  QEST nD: n-dimensional extension and generalisation of a software performance measurement model , 2002 .

[39]  Barry W. Boehm,et al.  Get Ready for Agile Methods, with Care , 2002, Computer.

[40]  Jing Tian,et al.  A Survey on Knowledge Management in Software Engineering , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security - Companion.

[41]  Tracy Hall,et al.  Motivators of Software Process Improvement: an analysis of practitioners' views , 2002, Journal of Systems and Software.

[42]  Paul R. Niven,et al.  Balanced Scorecard Step-by-Step: Maximizing Performance and Maintaining Results , 2002 .

[43]  Shahla Ghobadi,et al.  What drives knowledge sharing in software development teams: A literature review and classification framework , 2015, Inf. Manag..

[44]  Alain Abran,et al.  Introducing root-cause analysis and orthogonal defect classification at lower CMMI maturity levels , 2006 .

[45]  Jonathan Chard,et al.  A measurement maturity model , 1993, Software Quality Journal.

[46]  Touseef Tahir,et al.  Systematic Literature Review on Software Measurement Programs , 2016 .

[47]  Patricia J. Guinan,et al.  Enabling Software Development Team Performance During Requirements Definition: A Behavioral Versus Technical Approach , 1998, Inf. Syst. Res..

[48]  Robert H Yacobellis,et al.  A Method for Assessing Software Measurement Technology , 1990 .

[49]  Mohammad Alshayeb,et al.  An empirical study of system design instability metric and design evolution in an agile software process , 2005, J. Syst. Softw..

[50]  Laurie A. Williams,et al.  Assessing test-driven development at IBM , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[51]  D. Simpson,et al.  Assessing organizational readiness for change. , 2002, Journal of substance abuse treatment.

[52]  Joyce J. Elam,et al.  Inside a software design team: knowledge acquisition, sharing, and integration , 1993, CACM.

[53]  Tore Dybå,et al.  An empirical investigation of the key factors for success in software process improvement , 2005, IEEE Transactions on Software Engineering.

[54]  Manoj Kumar Tiwari,et al.  Modeling the metrics of lean, agile and leagile supply chain: An ANP-based approach , 2006, Eur. J. Oper. Res..