Performance-Driven Interface Contract Enforcement for Scientific Components

Several performance-driven approaches to selectively enforce interface contracts for scientific components are investigated. The goal is to facilitate debugging deployed applications built from plug-and-play components while keeping the cost of enforcement within acceptable overhead limits. This paper describes a study of global enforcement using a priori execution cost estimates obtained from traces. Thirteen trials are formed from five, single-component programs. Enforcement experiments conducted using twenty-three enforcement policies are used to determine the nature of exercised contracts and the impact of a variety of sampling strategies. Performance-driven enforcement appears to be best suited to programs that exercise moderately expensive contracts.

[1]  Jean-Marc Jézéquel,et al.  Making Components Contract Aware , 1999, Computer.

[2]  D. L. Parnas,et al.  On the criteria to be used in decomposing systems into modules , 1972, Software Pioneers.

[3]  James R. Larus,et al.  Using Paths to Measure, Explain, and Enhance Program Behavior , 2000, Computer.

[4]  Hong Sun,et al.  Investigating the use of analysis contracts to support fault isolation in object oriented code , 2002, ISSTA '02.

[5]  Wen-Tsuen Chen,et al.  Dynamic validation of programs using assertion checking facilities , 1978, COMPSAC.

[6]  Les Hatton,et al.  The T-experiments: errors in scientific software , 1996, Quality of Numerical Software.

[7]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[8]  Mike A. Marin,et al.  Effective use of assertions in C++ , 1996, SIGP.

[9]  Babak Hamidzadeh,et al.  Dynamic Task Scheduling Using Online Optimization , 2000, IEEE Trans. Parallel Distributed Syst..

[10]  Philippe Collet,et al.  Enforcing Different Contracts in Hierarchical Component-Based Systems , 2006, SC@ETAPS.

[11]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[12]  James Armstrong,et al.  Assertional checking and symbolic execution: An effective combination for debugging , 1979, ACM '79.

[13]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[14]  Lillian N. Cassel,et al.  Management of sampled real-time network measurements , 1989, [1989] Proceedings. 14th Conference on Local Computer Networks.

[15]  Stephen H. Edwards,et al.  A framework for detecting interface violations in component-based software , 1998, Proceedings. Fifth International Conference on Software Reuse (Cat. No.98TB100203).

[16]  John Franco.  The brick wall: NP completeness , 1997 .

[17]  Mary Shaw,et al.  Truth vs. knowledge: the difference between what a component does and what we know it does , 1996, Proceedings of the 8th International Workshop on Software Specification and Design.

[18]  Robyn R. Lutz,et al.  Empirical analysis of safety-critical anomalies during operations , 2004, IEEE Transactions on Software Engineering.

[19]  K.H. Lee,et al.  A request scheduling algorithm to support flexible resource reservations in advance , 2004, Canadian Conference on Electrical and Computer Engineering 2004 (IEEE Cat. No.04CH37513).

[20]  Dave Thomas.  Agile Programming: Design to Accommodate Change , 2005, IEEE Softw..

[21]  Robert W. Floyd,et al.  Assigning Meanings to Programs , 1993 .

[22]  Matthias Hauswirth,et al.  Low-overhead memory leak detection using adaptive statistical profiling , 2004, ASPLOS XI.

[23]  Eugene H. Spafford,et al.  Failure and fault analysis for software debugging , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[24]  Damien Watkins,et al.  Adding semantics to interface definition languages , 1998, Proceedings 1998 Australian Software Engineering Conference (Cat. No.98EX233).

[25]  Benoit Baudry,et al.  From testing to diagnosis: an automated approach , 2004 .

[26]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[27]  Bertrand Meyer,et al.  Object-Oriented Software Construction, 2nd Edition , 1997 .

[28]  Pedro Guerreiro.  Another mediocre assertion mechanism for C++ , 2000, Proceedings 33rd International Conference on Technology of Object-Oriented Languages and Systems TOOLS 33.

[29]  Ted J. Biggerstaff.  Reuse technologies and their niches , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[30]  Shari Lawrence Pfleeger,et al.  Software Quality: The Elusive Target , 1996, IEEE Softw..

[31]  Sebastian G. Elbaum,et al.  Anomalies as precursors of field failures , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[32]  João Gabriel Silva,et al.  Algorithm based fault tolerance versus result-checking for matrix computations , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[33]  Cynthia Della Torre Cicalese,et al.  Behavioral Specification of Distributed Software Component Interfaces , 1999, Computer.

[34]  Johann Rost.  Software Engineering Theory in Practice , 2005, IEEE Softw..

[35]  Jeffrey C. Carver,et al.  observations about Software development for High End Computing , 2006 .

[36]  Albert Endres.  An analysis of errors and their causes in system programs , 1975 .

[37]  Christian Blum,et al.  Metaheuristics in combinatorial optimization: Overview and conceptual comparison , 2003, CSUR.

[38]  David Lorge Parnas.  A technique for software module specification with examples , 1983, CACM.

[39]  Brent Hailpern,et al.  Software debugging, testing, and verification , 2002, IBM Syst. J..

[40]  Stuart Kent,et al.  Interpreting the object constraint language , 1998, Proceedings 1998 Asia Pacific Software Engineering Conference (Cat. No.98EX240).

[41]  Gary T. Leavens,et al.  JML: notations and tools supporting detailed design in Java , 2000 .

[42]  Matthias Felleisen,et al.  Contract Soundness for object-oriented languages , 2001, OOPSLA '01.

[43]  B. Curtis,et al.  Measurement and experimentation in software engineering , 1980, Proceedings of the IEEE.

[44]  C. A. R. HOARE,et al.  An axiomatic basis for computer programming , 1969, CACM.

[45]  C. A. R. Hoare,et al.  The emperor's old clothes , 1981, CACM.

[46]  Premkumar T. Devanbu,et al.  Improving scientific software component quality through assertions , 2005, SE-HPCS '05.

[47]  Edgar M. Pass,et al.  An adaptive microscheduler for a multiprogrammed computer system , 1973, ACM Annual Conference.

[48]  Les Hatton,et al.  Reexamining the Fault Density-Component Size Connection , 1997, IEEE Softw..

[49]  Edward B. Allen,et al.  Predicting risky modules in open-source software for high-performance computing , 2005, SE-HPCS '05.

[50]  David S. Reiner,et al.  A method for adaptive performance improvement of operating systems , 1981, SIGMETRICS '81.

[51]  Kinji Ono,et al.  Cost Estimation of User-Defined Methods in Advanced Database Systems , 1999 .

[52]  Wolfram Schulte,et al.  The ABCs of specification: asml, behavior, and components , 2001, Informatica.

[53]  Thomas E. Hull,et al.  Exception handling in scientific computing , 1988, TOMS.

[54]  Mark E. M. Stewart,et al.  An experiment in automated, scientific-code semantic analysis , 1999 .

[55]  Elaine J. Weyuker,et al.  The distribution of faults in a large industrial software system , 2002, ISSTA '02.

[56]  Nachiappan Nagappan.  Toward a software testing and reliability early warning metric suite , 2004, Proceedings. 26th International Conference on Software Engineering.

[57]  Richard N. Taylor.  Assertions in programming languages , 1980, SIGP.

[58]  Jean-Marc Jézéquel,et al.  Robustness and diagnosability of OO systems designed by contracts , 2001, Proceedings Seventh International Software Metrics Symposium.

[59]  Abdelwahab Hamou-Lhadj,et al.  SEAT: a usable trace analysis tool , 2005, 13th International Workshop on Program Comprehension (IWPC'05).

[60]  Kenneth N. Meyer,et al.  Evaluating COTS component dependability in context , 2005, IEEE Software.

[61]  James Arthur Kohl,et al.  Component-based software for high-performance scientific computing , 2005 .

[62]  Victor R. Basili,et al.  Software errors and complexity: an empirical investigation0 , 1984, CACM.

[63]  Behrooz Parhami,et al.  From defects to failures: a view of dependable computing , 1988, CARN.

[64]  Theresa L. Windus,et al.  Components for integral evaluation in quantum chemistry , 2008, J. Comput. Chem..

[65]  Shari Lawrence Pfleeger,et al.  Soup or Art? The Role of Evidential Force in Empirical Software Engineering , 2005, IEEE Softw..

[66]  Cristina Boeres,et al.  Hybrid task scheduling: integrating static and dynamic heuristics , 2003, Proceedings. 15th Symposium on Computer Architecture and High Performance Computing.

[67]  Philippe Collet,et al.  Towards efficient support for executing the Object Constraint Language , 1999, Proceedings of Technology of Object-Oriented Languages and Systems - TOOLS 30 (Cat. No.PR00278).

[68]  Premkumar T. Devanbu,et al.  Adaptable Assertion Checking for Scientific Software Components , 2004 .

[69]  Barry W. Boehm,et al.  COTS-Based Systems Top 10 List , 2001, Computer.

[70]  Paul F. Dubois,et al.  Maintaining correctness in scientific programs , 2005, Comput. Sci. Eng..

[71]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[72]  Patricia Adams,et al.  Programming Languages: Principles and Practice , 1993 .

[73]  Dorothy M. Andrews,et al.  An automated program testing methodology and its implementation , 1981, ICSE '81.

[74]  Jeffrey M. Voas.  Software Quality's Eight Greatest Myths , 1999, IEEE Softw..

[75]  Mark Sullivan,et al.  Software defects and their impact on system availability-a study of field failures in operating systems , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[76]  簡聰富,et al.  物件導向軟體之架構(Object-Oriented Software Construction)探討 , 1989 .

[77]  John C. Munson,et al.  Investigating software failures with a software black box , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[78]  David S. Rosenblum.  A Practical Approach to Programming With Assertions , 1995, IEEE Trans. Software Eng..

[79]  M. Young,et al.  Rethinking the Taxonomy of Fault Detection Techniques , 1989, 11th International Conference on Software Engineering.

[80]  S. H. Saib.  Executable Assertions - An Aid To Reliable Software , 1977 .

[81]  J. W. Hutchinson,et al.  A preliminary study of large-scale software re-use , 1988, Softw. Eng. J..

[82]  Victor R. Basili,et al.  Software errors and complexity: an empirical investigation , 1993 .

[83]  kc claffy,et al.  Application of sampling methodologies to network traffic characterization , 1993, SIGCOMM 1993.

[84]  Michael I. Jordan,et al.  Sampling User Executions for Bug Isolation , 2003 .

[85]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[86]  Daniel S. Katz,et al.  Tests and Tolerances for High-Performance Software-Implemented Fault Detection , 2003, IEEE Trans. Computers.

[87]  Ko-Yang Wang.  Precise compile-time performance prediction for superscalar-based computers , 1994, PLDI '94.

[88]  C. A. R. Hoare,et al.  Assertions: A Personal Perspective , 2003, IEEE Ann. Hist. Comput..

[89]  Keith H. Bennett,et al.  Legacy Systems: Coping with Success , 1995, IEEE Softw..

[90]  Raymond R. Hill.  An analytical comparison of optimization problem generation methodologies , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[91]  Andreas Drexl,et al.  A simulated annealing approach to the multiconstraint zero-one knapsack problem , 1988, Computing.

[92]  Mary Shaw,et al.  Empirical evaluation of defect projection models for widely-deployed production software systems , 2004, SIGSOFT '04/FSE-12.

[93]  Barry W. Boehm,et al.  What we have learned about fighting defects , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[94]  하수철,et al.  [서평]「Component Software」 - Beyond Object-Oriented Programming - , 2000 .

[95]  Elaine J. Weyuker,et al.  Testing Component-Based Software: A Cautionary Tale , 1998, IEEE Softw..

[96]  Robert L. Glass.  A Sad SAC Story about the State of the Practice , 2005, IEEE Softw..

[97]  Farnam Jahanian,et al.  An approach to monitoring and assertion-checking of real-time specifications , 1996, Proceedings of the 4th International Workshop on Parallel and Distributed Real-Time Systems.

[98]  Shari Lawrence Pfleeger,et al.  Investigating the Influence of Formal Methods , 1997, Computer.

[99]  Dewayne E. Perry,et al.  Software Faults in Evolving a Large, Real-Time System: a Case Study , 1993, ESEC.

[100]  Andreas Zeller,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[101]  David C. Luckham,et al.  An Overview of Anna, a Specification Language for Ada , 1985, IEEE Software.

[102]  James Arthur Kohl,et al.  A Component Architecture for High-Performance Scientific Computing , 2006, Int. J. High Perform. Comput. Appl..

[103]  D. W. Cooper.  Adaptive testing , 1976, ICSE '76.

[104]  David Abramson,et al.  Design and implementation of assertions for the common language infrastructure , 2003, IEE Proc. Softw..

[105]  Danny B. Lange,et al.  Object-Oriented Program Tracing and Visualization , 1997, Computer.

[106]  S. Flake.  Real-time constraints with the OCL , 2002, Proceedings Fifth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing. ISIRC 2002.

[107]  Timothy J. Tautges,et al.  The TSTT Mesh Interface , 2006 .

[108]  Reidar Conradi,et al.  An empirical study of software reuse vs. defect-density and stability , 2004, Proceedings. 26th International Conference on Software Engineering.

[109]  Robert W. Sebesta,et al.  Concepts of programming languages , 1973 .

[110]  Ali Hamie.  Enhancing the Object Constraint Language for more expressive specifications , 1999, Proceedings Sixth Asia Pacific Software Engineering Conference (ASPEC'99) (Cat. No.PR00509).

[111]  Elaine J. Weyuker,et al.  Where the bugs are , 2004, ISSTA '04.

[112]  David Leon,et al.  Pursuing failure: the distribution of program failures in a profile space , 2001, ESEC/FSE-9.

[113]  Barry W. Boehm,et al.  Software Defect Reduction Top 10 List , 2001, Computer.

[114]  Taghi M. Khoshgoftaar,et al.  Early Quality Prediction: A Case Study in Telecommunications , 1996, IEEE Softw..

[115]  Andy Roberts,et al.  How Accurate Is Scientific Software? , 1994, IEEE Trans. Software Eng..

[116]  Leslie Hatton,et al.  Software failures-follies and fallacies , 1997 .

[117]  Mark Fleischer.  Simulated annealing: past, present, and future , 1995, WSC '95.

[118]  George T. Heineman.  Integrating Interface Assertion Checkers into Component Models , 2003 .

[119]  W. V. Kozaczynski,et al.  Concern-driven design for a specification language supporting component-based software engineering , 1996, Proceedings of the 8th International Workshop on Software Specification and Design.

[120]  Abdelwahab Hamou-Lhadj,et al.  A survey of trace exploration tools and techniques , 2004, CASCON.

[121]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[122]  R. Kramer.  iContract - The Java(tm) Design by Contract(tm) Tool , 1998 .

[123]  Damien Watkins,et al.  Using interface definition languages to support path expressions and programming by contract , 1998, Proceedings. Technology of Object-Oriented Languages. TOOLS 26 (Cat. No.98EX176).

[124]  Ragnhild Van Der Straeten,et al.  Specifying and implementing the operational use of constraints in object-oriented applications , 2002 .

[125]  Changcheng Huang,et al.  Adaptive sampling methods for network performance metrics measurement and evaluation in MPLS-based IP networks , 2003, CCECE 2003 - Canadian Conference on Electrical and Computer Engineering. Toward a Caring and Humane Technology (Cat. No.03CH37436).

[126]  John C. Munson,et al.  Toward a quantifiable definition of software faults , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[127]  Thomas L. Casavant,et al.  A Taxonomy of Scheduling in General-Purpose Distributed Computing Systems , 1988, IEEE Trans. Software Eng..

[128]  Hajime Kita,et al.  Genetic algorithms for adaptation to dynamic environments - a survey , 2000, 2000 26th Annual Conference of the IEEE Industrial Electronics Society. IECON 2000. 2000 IEEE International Conference on Industrial Electronics, Control and Instrumentation. 21st Century Technologies.

[129]  Bertrand Meyer,et al.  The grand challenge of trusted components , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[130]  Donatella Sciuto,et al.  Library functions timing characterization for source-level analysis , 2003, 2003 Design, Automation and Test in Europe Conference and Exhibition.

[131]  Paul F. Dubois.  Scientific Components Are Coming , 1999, Computer.

[132]  Jin-Kao Hao,et al.  A hybrid approach for the 0-1 multidimensional knapsack problem , 2001, IJCAI 2001.

[133]  E. L. Harder,et al.  The Institute of Electrical and Electronics Engineers, Inc. , 2019, 2019 IEEE International Conference on Software Architecture Companion (ICSA-C).

[134]  Jeffrey M. Voas.  Quality Time: How Assertions Can Increase Test Effectiveness , 1997, IEEE Softw..

[135]  Otthein Herzog,et al.  SLAN-4-A software specification and design language , 1980, IEEE Transactions on Software Engineering.

[136]  Allen D. Malony,et al.  Computational Quality of Service for Scientific CCA Applications: Composition, Substitution, and Reconfiguration , 2006 .

[137]  Brian W. Kernighan,et al.  The C Programming Language , 1978 .

[138]  Ralph L. London,et al.  A view of program verification , 1975, Reliable Software.

[139]  Peter Winkler.  Optimality and Greed in Dynamic Allocation , 2001, J. Algorithms.

[140]  Reinhold Plösch.  Design by Contract for Python , 1997, APSEC.

[141]  Sebastian G. Elbaum,et al.  Software reliability as a function of user execution patterns , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[142]  Warren Harrison.  Skinner Wasn't a Software Engineer , 2005, IEEE Softw..

[143]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[144]  Wojtek Kozaczynski,et al.  Toward software plug-and-play , 1997, SSR '97.

[145]  David K. Gifford,et al.  Static dependent costs for estimating execution time , 1994, LFP '94.

[146]  James R. Larus,et al.  Righting software , 2004, IEEE Software.

[147]  Richard Mitchell,et al.  Contract-oriented specifications , 1997, Proceedings. Technology of Object-Oriented Languages. TOOLS 24 (Cat. No.97TB100240).

[148]  James Arthur Kohl,et al.  Parallel PDE-Based Simulations Using the Common Component Architecture , 2006 .

[149]  Dewayne E. Perry,et al.  EMPIRICAL STUDY OF SOFTWARE INTERFACE FAULTS. , 1985 .

[150]  Barry Boehm,et al.  Top 10 list [software development] , 2001 .

[151]  Matthew Arnold,et al.  A framework for reducing the cost of instrumented code , 2001, PLDI '01.

[152]  Sriram Sankar,et al.  ADL—an interface definition language for specifying and testing software , 1994 .

[153]  Albert Endres.  Lessons learned in an industrial software lab (software development) , 1993, IEEE Software.

[154]  Robert L. Glass,et al.  Focusing on the application domain: everyone agrees it's vital, but who's doing anything about it? , 1998, Proceedings of the Thirty-First Hawaii International Conference on System Sciences.

[155]  Reinhold Plösch,et al.  Evaluation of Assertion Support for the Java Programming Language , 2002, J. Object Technol..

[156]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[157]  Carlos Gonzalez,et al.  A model for an adaptive scheduler , 1988, CSC '88.