IR 8101 i A Rational Foundation for Software Metrology

Much software research and practice involves ostensible measurements of software, yet little progress has been made on a metrological foundation like the International System of Units (SI) for those measurements since the work of Gray, Hogan, et al. in 1996–2001. Given a physical object, one can determine physical properties using measurement principles and express measured values using standard quantities that have concrete realizations. In contrast, most software metrics are simple counts that are used as indicators of complex, abstract qualities. In this report we revisit software metrology from two directions: first, top down, to establish a theory of software measurement; second, bottom up, to identify specific purposes for which software measurements are needed, quantifiable properties of software, relevant units, and objects of measurement. Although there are structural obstacles to realizing the vision of software metrology that works like physical metrology for all desired measurands, progress is possible if we start with a rational foundation.

[1]  P. Jaccard,et al.  Etude comparative de la distribution florale dans une portion des Alpes et des Jura , 1901 .

[2]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[3]  Patrick Suppes,et al.  Foundations of measurement , 1971 .

[4]  U. Grenander,et al.  Quantitative Methods for Evaluating Computer System Performance: a Review and Proposals , 1972, Statistical Computer Performance Evaluation.

[5]  Fred P. Brooks,et al.  The Mythical Man-Month , 1975, Reliable Software.

[6]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[7]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[8]  Philip B. Crosby,et al.  Quality Is Free: The Art of Making Quality Certain , 1979 .

[9]  Sallie M. Henry,et al.  Software Structure Metrics Based on Information Flow , 1981, IEEE Transactions on Software Engineering.

[10]  John A. Simpson,et al.  Foundations of Metrology. , 1981, Journal of research of the National Bureau of Standards.

[11]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[12]  Boris Beizer,et al.  Software System Testing and Quality Assurance , 1984 .

[13]  B. Fischhoff,et al.  Assessing uncertainty in physical constants , 1986 .

[14]  Robert F. Berry.  Computer Benchmark Evaluation and Design of Experiments, a Case Study , 1992, IEEE Trans. Computers.

[15]  P. De Biévre,et al.  'Atomic weight': The name, its history, definition, and units , 1992 .

[16]  Leonard R. Sussman,et al.  Nominal, Ordinal, Interval, and Ratio Typologies are Misleading , 1993 .

[17]  T. Hopp.  Computational Metrology , 1993 .

[18]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[19]  Wayne J. Salamon,et al.  Quality Characteristics and Metrics for Reusable Software (Preliminary Report) | NIST , 1994 .

[20]  James M. Bieman,et al.  Cohesion and reuse in an object-oriented system , 1995, SSR '95.

[21]  Martin Hitz,et al.  Measuring coupling and cohesion in object-oriented systems , 1995 .

[22]  Tsutomu Ishida,et al.  Metrics and Models in Software Quality Engineering , 1995 .

[23]  Michael A. Johnson,et al.  Effective and Appropriate Use of Controlled Experi-mentation in Software Development Research , 1996 .

[24]  Dolores R. Wallace,et al.  Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric , 1996 .

[25]  Lutz E. Schlange.  Scenarios: The art of strategic conversation , 1997 .

[26]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[27]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[28]  Thomas J. Mowbray,et al.  AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis , 1998 .

[29]  Martha M. Gray,et al.  Applicability of Metrology to Information Technology , 1999, Journal of Research of the National Institute of Standards and Technology.

[30]  James E. Fowler,et al.  Information Technology Measurement and Testing Activities at NIST , 2001, Journal of research of the National Institute of Standards and Technology.

[31]  A. O'Hagan,et al.  Bayesian calibration of computer models , 2001 .

[32]  Cem Kaner,et al.  Software Engineering Metrics: What Do They Measure and How Do We Know? , 2004 .

[33]  Clifton A. Ericson,et al.  Hazard Analysis Techniques for System Safety , 2005 .

[34]  Eswaran Subrahmanian,et al.  Evaluating Reasoning Systems , 2006 .

[35]  Naji Habra,et al.  ISO-based model to measure software product quality , 2007 .

[36]  R. Atkinson,et al.  Digital Prosperity: Understanding the Economic Benefits of the Information Technology Revolution , 2007 .

[37]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[38]  Matthias Hauswirth,et al.  Producing wrong data without doing anything obviously wrong! , 2009, ASPLOS.

[39]  Ken Pugh.  Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration , 2010 .

[40]  Alain Abran,et al.  Software Metrics and Software Metrology , 2010 .

[41]  Elizabeth N. Fong,et al.  Toward a Preliminary Framework for Assessing the Trustworthiness of Software , 2010 .

[42]  Lionel C. Briand,et al.  An Industrial Investigation of Similarity Measures for Model-Based Test Case Selection , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[43]  Fabian Beck,et al.  On the congruence of modularity and code coupling , 2011, ESEC/FSE '11.

[44]  Paul E. Black,et al.  Counting Bugs is Harder Than You Think , 2011, 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation.

[45]  R. White,et al.  The meaning of measurement in metrology , 2011 .

[46]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[47]  Stephen L. R. Ellison,et al.  Dark uncertainty , 2011 .

[48]  A. Kostinski,et al.  Bounds on spectral dispersion from Fermi-detected gamma ray bursts. , 2011, Physical review letters.

[49]  René Dybkaer,et al.  Revision of the ‘Guide to the Expression of Uncertainty in Measurement’. Why and how , 2012 .

[50]  Jan Vitek,et al.  R3: repeatability, reproducibility and rigor , 2012, SIGP.

[51]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-wise Test Suites for Large Software Product Lines , 2012, ArXiv.

[52]  David W. Flater,et al.  Configuration of profiling tools for C/C++ applications under 64-bit Linux , 2013 .

[53]  Tomas Kalibera,et al.  Rigorous benchmarking in reasonable time , 2013, ISMM '13.

[54]  Jeff Yu Lei,et al.  Combinatorial Coverage Measurement Concepts and Applications , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[55]  W. Bich.  Revision of the ?Guide to the Expression of Uncertainty in Measurement?. Why and how , 2014 .

[56]  Sushil Jajodia,et al.  k-Zero Day Safety: A Network Security Metric for Measuring the Risk of Unknown Vulnerabilities , 2014, IEEE Transactions on Dependable and Secure Computing.

[57]  David W. Flater,et al.  The ghost in the machine: Don't let it haunt your software performance measurements , 2014 .

[58]  D. Flater.  Screening for factors affecting application performance in profiling measurements , 2014 .

[59]  John C. Knight,et al.  A security metric based on security arguments , 2014, WETSoM 2014.

[60]  R. Kacker.  Probability distributions and coverage probability in GUM, JCGM documents, and statistical inference , 2015 .

[61]  P. Mohr,et al.  Reply to Comments on ‘Dimensionless units in the SI’ , 2015 .

[62]  P. Mohr,et al.  Dimensionless units in the SI , 2014, 1409.2794.

[63]  Sayed Mehran Sharafi,et al.  A METRIC -BASED APPROACH FOR MEASURING THE CONCEPTUAL INTEGRITY OF SOFTWARE ARCHITECTURES , 2015 .

[64]  Jan Vitek.  Repeatability, reproducibility and rigor in CS research , 2015, PLMW '15.

[65]  S. Pommé,et al.  Uncertainty of nuclear counting , 2015 .

[66]  B. P. Leonard,et al.  Comment on ‘Dimensionless units in the SI’ , 2015 .

[67]  Doreen Meier,et al.  Structured Design Fundamentals Of A Discipline Of Computer Program And Systems Design , 2016 .