Unveiling anomalies and their impact on software quality in model-based automotive software revisions with software metrics and domain experts

The validation of simulation models (e.g., of electronic control units for vehicles) in industry is becoming increasingly challenging due to their growing complexity. To systematically assess the quality of such models, software metrics seem to be promising. In this paper we explore the use of software metrics and outlier analysis as a means to assess the quality of model-based software. More specifically, we investigate how results from regression analysis applied to measurement data received from size and complexity metrics can be mapped to software quality. Using the moving averages approach, models were fit to data received from over 65,000 software revisions for 71 simulation models that represent different electronic control units of real premium vehicles. Consecutive investigations using studentized deleted residuals and Cook’s Distance revealed outliers among the measurements. From these outliers we identified a subset, which provides meaningful information (anomalies) by comparing outlier scores with expert opinions. Eight engineers were interviewed separately for outlier impact on software quality. Findings were validated in consecutive workshops. The results show correlations between outliers and their impact on four of the considered quality characteristics. They also demonstrate the applicability of this approach in industry.

[1]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[2]  Joseph Gil,et al.  An empirical investigation of changes in some software properties over time , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[3]  Nicholas A. Kraft,et al.  Cyclomatic Complexity and Lines of Code: Empirical Evidence of a Stable Linear Relationship , 2009, J. Softw. Eng. Appl..

[4]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[5]  Jeremy Garnett,et al.  Business Statistics in Practice , 2003 .

[6]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[7]  Marina Waldén,et al.  Quality Comparison and Evaluation of Digital Hydraulic Control Systems , 2008 .

[8]  Oscar Pastor,et al.  Advanced Information Systems Engineering : 25th International Conference, CAiSE 2013, Valencia, Spain, June 17-21, 2013. Proceedings , 2013 .

[9]  Robert Feldt,et al.  Validity Threats in Empirical Software Engineering Research - An Initial Survey , 2010, SEKE.

[10]  Emad Shihab,et al.  Characterizing and predicting blocking bugs in open source projects , 2014, MSR 2014.

[11]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986, Encyclopedia of Big Data.

[12]  Elaine J. Weyuker,et al.  Evaluating Software Complexity Measures , 2010, IEEE Trans. Software Eng..

[13]  Aditya K. Ghose,et al.  Characterization and Prediction of Issue-Related Risks in Software Projects , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[14]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[15]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[16]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[17]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[18]  Andreas Zeller,et al.  Predicting defects using change genealogies , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[19]  Sandro Morasca,et al.  Property-Based Software Engineering Measurement , 1996, IEEE Trans. Software Eng..

[20]  Jens H. Weber,et al.  Predicting maintainability with object-oriented metrics -an empirical comparison , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[21]  David A. Gustafson,et al.  Properties of Software Measures , 1991, Formal Aspects of Measurement.

[22]  Miroslaw Staron,et al.  Comparing the Applicability of Complexity Measurements for Simulink Models during Integration Testing -- An Industrial Case Study , 2015, 2015 IEEE/ACM 2nd International Workshop on Software Architecture and Metrics.

[23]  David N. Card,et al.  Measuring software design complexity , 1988, J. Syst. Softw..

[24]  James M. Bieman,et al.  Software Metrics: A Rigorous and Practical Approach, Third Edition , 2014 .

[25]  Christian Berger,et al.  Challenges from integration testing using interconnected hardware-in-the-loop test rigs at an automotive OEM – an industrial experience report , 2015, 2015 First International Workshop on Automotive Software Architecture (WASA).

[26]  D P Hartmann,et al.  Interrupted time-series analysis and its application to behavioral data. , 1980, Journal of applied behavior analysis.

[27]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[28]  Taghi M. Khoshgoftaar,et al.  Improving code churn predictions during the system test and maintenance phases , 1994, Proceedings 1994 International Conference on Software Maintenance.

[29]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data: A Survey , 2014, IEEE Transactions on Knowledge and Data Engineering.

[30]  Moe Thandar Wynn,et al.  Profiling Event Logs to Configure Risk Indicators for Process Delays , 2013, CAiSE.

[31]  Oral Alan,et al.  Thresholds based outlier detection approach for mining class outliers: An empirical case study on software measurement datasets , 2011, Expert Syst. Appl..