Replication of Controlled Experiments in Empirical Software Engineering - A Survey

The objective of this thesis is to collect guidelines for the replication of software engineering experiments, and to refine and supplement these guidelines where needed. These guidelines will hopefully contribute to the establishment of an empirical framework for the software engineering discipline.

[1]  D. Ross Jeffery,et al.  Early Lifecycle Work: Influence of Individual Characteristics, Methodological Constraints, and Interface Constraints , 2004, Empirical Software Engineering.

[2]  Padmal Vitharana,et al.  Computer-Mediated Group Support, Anonymity, and the Software Inspection Process: An Empirical Investigation , 2003, IEEE Trans. Software Eng..

[3]  Oliver Laitenberger,et al.  Cost-effective Detection of Software Defects through Perspective-based Inspections , 2001, Empirical Software Engineering.

[4]  Victor R. Basili,et al.  Experimentation in software engineering , 1986, IEEE Transactions on Software Engineering.

[5]  John C. Knight,et al.  Requirements : A Linguistic Basis for Their Occurrence and an Approach to Their Reduction , 2022 .

[6]  Christof Ebert,et al.  e-R&D – Effectively Managing Process Diversity , 2002, Ann. Softw. Eng..

[7]  Jordan Dimitrov.  Operational semantics for Verilog , 2001, Proceedings Eighth Asia-Pacific Software Engineering Conference.

[8]  P. Mellor.  A model of the problem or a problem with the model , 1998 .

[9]  Adam A. Porter,et al.  Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects , 1998, Empirical Software Engineering.

[10]  Sumin Huang,et al.  Developing distributed applications by semantics-based automatic replication , 1994, Proceedings of 1st Asia-Pacific Software Engineering Conference.

[11]  Christopher M. Lott,et al.  Repeatable software engineering experiments for comparing defect-detection techniques , 2004, Empirical Software Engineering.

[12]  Jane Grimson,et al.  A distributed version control system for wide area networks , 1990, Softw. Eng. J..

[13]  Rachid Guerraoui,et al.  e-Transactions: End-to-End Reliability for Three-Tier Architectures , 2002, IEEE Trans. Software Eng..

[14]  Ove Hansen.  Survey of Controlled Software Engineering Experiments with Focus on Subjects , 2004 .

[15]  A. J. Hoane,et al.  Design and Analysis in Software Engineering Part 1 : The Language of Case Studies and Formal Experiments , 2004 .

[16]  T. C. Nicholas Graham,et al.  Dragonfly: linking conceptual and implementation architectures of multiuser interactive systems , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[17]  William M. Evanco,et al.  Comments on "The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics" , 2003, IEEE Trans. Software Eng..

[18]  Chi-Kwong Li,et al.  Strategic selection and replication of movies by trend-calibrated movie-demand model , 2000, Proceedings International Symposium on Multimedia Software Engineering.

[19]  Shi-Kuo Chang.  Multimedia Software Engineering , 2000, International Series in Software Engineering.

[20]  Per Runeson,et al.  Are the Perspectives Really Different? – Further Experimentation on Scenario-Based Reading of Requirements , 2000, Empirical Software Engineering.

[21]  Giuliana Franceschinis,et al.  Parametric Fault Tree for the Dependability Analysis of Redundant Systems and Its High-Level Petri Net Semantics , 2003, IEEE Trans. Software Eng..

[22]  L. Delbeke.  Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[23]  Giovanni Cantone,et al.  Comparing code reading techniques applied to object-oriented software frameworks with regard to effectiveness and defect detection rate , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[24]  Jeff Magee,et al.  Client Access Protocols for Replicated Services , 1999, IEEE Trans. Software Eng..

[25]  Audris Mockus,et al.  A Web laboratory for software data analysis , 2004, World Wide Web.

[26]  Yasuyuki Tsukada,et al.  Interactive and Probabilistic Proof of Mobile Code Safety , 2005, Automated Software Engineering.

[27]  Ian Gorton,et al.  A predictive performance model to evaluate the contention cost in application servers , 2002, Ninth Asia-Pacific Software Engineering Conference, 2002..

[28]  Ing-Ray Chen,et al.  User-perceived availability and response-time in voting-based replicated systems: a case study , 1999, Proceedings 1999 IEEE Symposium on Application-Specific Systems and Software Engineering and Technology. ASSET'99 (Cat. No.PR00122).

[29]  Zahira Ammarguellat,et al.  A Control-Flow Normalization Algorithm and Its Complexity , 1992, IEEE Trans. Software Eng..

[30]  Tore Dybå,et al.  Conducting realistic experiments in software engineering , 2002, Proceedings International Symposium on Empirical Software Engineering.

[31]  Satish K. Tripathi,et al.  Capacity of Voting Systems , 1993, IEEE Trans. Software Eng..

[32]  A. Brooks,et al.  Verification of results in software maintenance through external replication , 1994, Proceedings 1994 International Conference on Software Maintenance.

[33]  Akira Fukuda,et al.  Design of protocols in Timed CSP for highly reliable and available client-server system , 1997, Proceedings of Joint 4th International Computer Science Conference and 4th Asia Pacific Software Engineering Conference.

[34]  El EmamKalhed,et al.  The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics , 2001 .

[35]  Mario Piattini,et al.  Finding "early" indicators of UML class diagrams understandability and modifiability , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[36]  Will Hayes,et al.  Research synthesis in software engineering: a case for meta-analysis , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[37]  Udi Manber,et al.  Concurrent Maintenance of Binary Search Trees , 1984, IEEE Transactions on Software Engineering.

[38]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[39]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[40]  Keith Phalp,et al.  Replicating the CREWS Use Case Authoring Guidelines Experiment , 2000, Empirical Software Engineering.

[41]  Sidney Addelman,et al.  trans-Dimethanolbis(1,1,1-trifluoro-5,5-dimethylhexane-2,4-dionato)zinc(II) , 2008, Acta crystallographica. Section E, Structure reports online.

[42]  Gianfranco Ciardo,et al.  Stochastic Petri Net Analysis of a Replicated File System , 1989, IEEE Trans. Software Eng..

[43]  Pekka Abrahamsson,et al.  Commitment development in software process improvement: critical misconceptions , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[44]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[45]  José Alves Marques,et al.  Development of distributed applications with separation of concerns , 1995, Proceedings 1995 Asia Pacific Software Engineering Conference.

[46]  Eliot R. Smith,et al.  Research methods in social relations , 1962 .

[47]  Shaoying Liu,et al.  A GUI and testing tool for SOFL , 2000, Proceedings Seventh Asia-Pacific Software Engeering Conference. APSEC 2000.

[48]  Nancy A. Lynch,et al.  Discarding Obsolete Information in a Replicated Database System , 1987, IEEE Transactions on Software Engineering.

[49]  M Roper.  Replication's Role in Experimental Computer Science , 1994 .

[50]  Mario Jino,et al.  Identification of potentially infeasible program paths by monitoring the search for test data , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[51]  Mostafa H. Ammar,et al.  Performance Characterization of Quorum-Consensus Algorithms for Replicated Data , 1989, IEEE Trans. Software Eng..

[52]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[53]  Magne Jørgensen,et al.  Regression Models of Software Development Effort Estimation Accuracy and Bias , 2004, Empirical Software Engineering.

[54]  Witold Suryn,et al.  New directions in measurement for software quality control , 2002, 10th International Workshop on Software Technology and Engineering Practice.

[55]  Nima Kaveh,et al.  Model checking distributed objects design , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[56]  G. Rochlin.  An information model for intelligent network services , 1989 .

[57]  Deron Liang,et al.  A nested invocation suppression mechanism for active replication fault-tolerant CORBA , 2002, Ninth Asia-Pacific Software Engineering Conference, 2002..

[58]  I. Nonaka,et al.  The Concept of “Ba”: Building a Foundation for Knowledge Creation , 1998 .

[59]  Clement T. Yu,et al.  Query Processing in a Fragmented Relational Distributed System: Mermaid , 1985, IEEE Transactions on Software Engineering.

[60]  Jürgen Münch,et al.  Using empirical knowledge from replicated experiments for software process simulation: a practical example , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[61]  Mukesh Singhal,et al.  Update Transport: A New Technique for Update Synchronization in Replicated Database Systems , 1990, IEEE Trans. Software Eng..

[62]  Adam A. Porter,et al.  Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment , 1995, IEEE Trans. Software Eng..

[63]  Koushik Sen,et al.  Efficient decentralized monitoring of safety in distributed systems , 2004, Proceedings. 26th International Conference on Software Engineering.

[64]  Chris Sauer,et al.  Validating the defect detection performance advantage of group designs for software reviews: report of a replicated experiment , 1997, Proceedings of Australian Software Engineering Conference ASWEC 97.

[65]  Khaled El Emam,et al.  The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics , 2001, IEEE Trans. Software Eng..

[66]  Bonnie Kaplan,et al.  Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study , 1988, MIS Q..

[67]  Witold Pedrycz,et al.  An Empirical Exploration of the Distributions of the Chidamber and Kemerer Object-Oriented Metrics Suite , 2004, Empirical Software Engineering.

[68]  Lionel C. Briand,et al.  Investigating quality factors in object-oriented designs: an industrial case study , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[69]  John W. Daly,et al.  Evaluating inheritance depth on the maintainability of object-oriented software , 2004, Empirical Software Engineering.

[70]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[71]  Sang Hyuk Son.  Semantic information and consistency in distributed realtime systems , 1988 .

[72]  John William Daly,et al.  Replication and a multi-method approach to empirical software engineering research , 1996 .

[73]  Miroslaw Staron,et al.  Empirical assessment of using stereotypes to improve comprehension of UML models: A set of experiments , 2006, J. Syst. Softw..

[74]  Per Runeson,et al.  Prospects and Limitations for Cross-Study Analyses – A Study on an Experiment Series , 2003 .

[75]  Philip S. Yu,et al.  Analysis of Concurrency-Coherency Control Protocols for Distributed Transaction Processing Systems with Regional Locality , 1992, IEEE Trans. Software Eng..

[76]  Janis Putman,et al.  General framework for fault tolerance from ISO/ITU Reference Model for Open Distributed Processing (RM-ODP) , 1999, Proceedings. Fifth International Workshop on Object-Oriented Real-Time Dependable Systems.

[77]  Dietmar Pfahl,et al.  An Externally Replicated Experiment for Evaluating the Learning Effectiveness of Using Simulations in Software Project Management Education , 2003, Empirical Software Engineering.

[78]  Khaled El Emam,et al.  Findings from Phase 2 of the SPICE trials , 2001, Softw. Process. Improv. Pract..

[79]  Calton Pu,et al.  Regeneration of replicated objects: A technique and its Eden implementation , 1986, 1986 IEEE Second International Conference on Data Engineering.

[80]  W WahBenjamin,et al.  Learning Dominance Relations in Combined Search Problems , 1988 .

[81]  Albert L. Baker,et al.  Synthesizing structured analysis and object‐based formal specifications , 1997, Ann. Softw. Eng..

[82]  Deron Liang,et al.  A nested invocation suppression mechanism for active replicated SOAP systems , 2004, 11th Asia-Pacific Software Engineering Conference.

[83]  Nachiappan Nagappan.  Toward a software testing and reliability early warning metric suite , 2004, Proceedings. 26th International Conference on Software Engineering.

[84]  Peter Triantafillou,et al.  Achieving Strong Consistency in a Distributed File System , 1997, IEEE Trans. Software Eng..

[85]  Stefan Biffl,et al.  Investigating the effect of expert ranking of use cases for design inspection , 2004, Proceedings. 30th Euromicro Conference, 2004..

[86]  Lionel C. Briand,et al.  Replicated Case Studies for Investigating Quality Factors in Object-Oriented Designs , 2001, Empirical Software Engineering.

[87]  A. Ehrenberg,et al.  The Design of Replicated Studies , 1993 .

[88]  John C. Knight.  Erroneous Requirements: A Linguistic Basis for Their Occurrence and an Approach to Their Reduction , 2001, HASE.

[89]  Kun-Lung Wu,et al.  Companson and Diagnosis of Large Replicated Files , 1987, IEEE Transactions on Software Engineering.

[90]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[91]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[92]  James R. Larus,et al.  Teapot: A Domain-Specific Language for Writing Cache Coherence Protocols , 1999, IEEE Trans. Software Eng..

[93]  M Wood,et al.  Replication of Experimental Results in Software Engineering , 2022 .

[94]  Michael Pidd,et al.  Critical issues in the development of component-based discrete simulation , 2004, Simul. Model. Pract. Theory.

[95]  Sevki S. Erdogan,et al.  An Installable Version Control File System for UNIX , 1997, Softw. Pract. Exp..

[96]  Emilia Mendes,et al.  A replicated assessment of the use of adaptation rules to improve Web cost estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[97]  Rainfield Y. Yen,et al.  Identification of a noiseless nonlinear system by gradient method , 2000, Proceedings International Symposium on Multimedia Software Engineering.

[98]  José Barata,et al.  Object oriented software engineering for programmable logical controllers: a successful implementation , 2003, EFTA 2003. 2003 IEEE Conference on Emerging Technologies and Factory Automation. Proceedings (Cat. No.03TH8696).

[99]  Satish K. Tripathi,et al.  The MARUTI hard real-time operating system , 1989, OPSR.

[100]  Giovanni Denaro,et al.  An empirical evaluation of fault-proneness models , 2002, ICSE '02.

[101]  I. Nonaka,et al.  BUILDING A FOUNDATION FOR KNOWLEDGE CREATION , 1998 .

[102]  Lionel C. Briand,et al.  A replicated assessment and comparison of common software cost modeling techniques , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[103]  Magne Jørgensen,et al.  Can you Trust a Single Data Source Exploratory Software Engineering Case Study? , 2002, Empirical Software Engineering.

[104]  R. Hayes,et al.  The Design of the Saguaro Distributed Operating System , 1987, IEEE Transactions on Software Engineering.

[105]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[106]  Lui Sha,et al.  Integration of CORBA services with a dynamic real-time architecture , 2000, Proceedings Sixth IEEE Real-Time Technology and Applications Symposium. RTAS 2000.

[107]  Andrew Taylor.  An operations perspective on strategic alliance success factors , 2005 .

[108]  Sushil Jajodia,et al.  A Pessimistic Consistency Control Algorithm for Replicated Files which Achieves High Availability , 1989, IEEE Trans. Software Eng..

[109]  Giuseppe Visaggio,et al.  A Replicated Experiment to Assess Requirements Inspection Techniques , 2004, Empirical Software Engineering.

[110]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[111]  J. Rushby,et al.  Formal verification of algorithms for critical systems , 1991, SIGSOFT '91.

[112]  Karl M. Göschka,et al.  Fault-tolerance in a distributed management system: a case study , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[113]  Anna Hác.  A Distributed Algorithm for Performance Improvement Through File Replication, File Migration and Process Migration , 1986, SIGMETRICS Perform. Evaluation Rev..

[114]  Volkmar Lotz,et al.  A Formal Security Model for Microprocessor Hardware , 1999, IEEE Trans. Software Eng..

[115]  Shari Lawrence Pfleeger,et al.  Design and analysis in software engineering: the language of case studies and formal experiments , 1994, SOEN.

[116]  Taghi M. Khoshgoftaar,et al.  Assessment of a New Three-Group Software Quality Classification Technique: An Empirical Case Study , 2005, Empirical Software Engineering.

[117]  Marin Litoiu,et al.  Designing Process Replication and Activation: A Quantitative Approach , 2000, IEEE Trans. Software Eng..

[118]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[119]  Pablo César,et al.  Open graphical framework for interactive TV , 2003, Fifth International Symposium on Multimedia Software Engineering, 2003. Proceedings..

[120]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[121]  P. Hu,et al.  Low storage cost, partition-tolerant dynamic algorithms for replicated file systems , 1992, CompEuro 1992 Proceedings Computer Systems and Software Engineering.

[122]  Tony Moynihan.  'Requirements-uncertainty': should it be a latent, aggregate or profile construct? , 2000, Proceedings 2000 Australian Software Engineering Conference.

[123]  Philip M. Johnson,et al.  Does Every Inspection Really Need a Meeting? , 1998, Empirical Software Engineering.

[124]  Zbigniew M. Wójcik,et al.  Sufficient Condition for a Communication Deadlock and Distributed Deadlock Detection , 1989, IEEE Trans. Software Eng..

[125]  Andreas Zendler,et al.  A Preliminary Software Engineering Theory as Investigated by Published Experiments , 2001, Empirical Software Engineering.

[126]  Jeffrey C. Carver,et al.  Knowledge-Sharing Issues in Experimental Software Engineering , 2004, Empirical Software Engineering.

[127]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[128]  Patrik Berander,et al.  Using students as subjects in requirements prioritization , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[129]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[130]  James Miller.  Statistical significance testing--a panacea for software technology experiments? , 2004, J. Syst. Softw..

[131]  Narain H. Gehani.  Concurrent C: real-time programming and fault tolerance , 1991, Softw. Eng. J..

[132]  Richard McClatchey,et al.  An Integrated Approach for Extraction of Objects From XML and Transformation to Heterogeneous Object Oriented Databases , 2003, ICEIS.

[133]  Nabil R. Adam,et al.  Regeneration with Virtual Copies for Distributed Computing Systems , 1993, IEEE Trans. Software Eng..

[134]  Danco Davcev,et al.  A Dynamic Voting Scheme in Distributed Systems , 1989, IEEE Trans. Software Eng..

[135]  Tetsuro Katayama.  Proposal of a supporting method for diagram generation with the transformation rules in UML , 2002, Ninth Asia-Pacific Software Engineering Conference, 2002..

[136]  Paul Lukowicz,et al.  Experimental evaluation in computer science: A quantitative study , 1995, J. Syst. Softw..

[137]  Daniel L. Moody,et al.  Evaluating the quality of information models: empirical testing of a conceptual model quality framework , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[138]  Khaled El Emam,et al.  Validating the ISO/IEC 15504 measures of software development process capability , 2000, J. Syst. Softw..

[139]  A. R. Ilersic,et al.  Research methods in social relations , 1961 .

[140]  Jeff Magee,et al.  A distributed file server for embedded applications , 1988, Softw. Eng. J..

[141]  David Eichmann,et al.  Supporting multiple domains in a single reuse repository , 1992, Proceedings Fourth International Conference on Software Engineering and Knowledge Engineering.

[142]  Michael Ogg,et al.  Experience with Distributed Replicated Objects: The Nile Project , 1998, Theory Pract. Object Syst..

[143]  Anders Wesslén,et al.  A Replicated Empirical Study of the Impact of the Methods in the PSP on Individual Engineers , 2000, Empirical Software Engineering.

[144]  Shaoying Liu.  A rigorous approach to reviewing formal specifications , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[145]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[146]  Michelle Cartwright,et al.  An empirical view of inheritance , 1998, Inf. Softw. Technol..

[147]  Jörg Rech,et al.  Using academic courses for empirical validation of software development processes , 2004, Proceedings. 30th Euromicro Conference, 2004..

[148]  Khaled El Emam,et al.  Validating the ISO/IEC 15504 Measure of Software Requirements Analysis Process Capability , 2000, IEEE Trans. Software Eng..

[149]  J. Diaz-Herrera,et al.  A Collaborative Program to Retrain Lockheed Martin Aero Engineers , 2002, IEEE Softw..

[150]  Boby George,et al.  A structured experiment of test-driven development , 2004, Inf. Softw. Technol..

[151]  Jeffrey C. Carver,et al.  Understanding the impact of assumptions on experimental validity , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[152]  Marvin V. Zelkowitz,et al.  Experimental validation in software engineering , 1997, Inf. Softw. Technol..

[153]  James Miller,et al.  Further Experiences with Scenarios and Checklists , 1998, Empirical Software Engineering.

[154]  Arthur J. Bernstein,et al.  An Application of Name Based Addressing to Low Level Distributed Algorithms , 1985, IEEE Transactions on Software Engineering.

[155]  Jeffrey C. Carver,et al.  Issues in using students in empirical studies in software engineering education , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[156]  Maarten van Steen,et al.  Software engineering for the scalable distributed applications , 1998, Proceedings. The Twenty-Second Annual International Computer Software and Applications Conference (Compsac '98) (Cat. No.98CB 36241).

[157]  Barbara A. Kitchenham,et al.  Combining empirical results in software engineering , 1998, Inf. Softw. Technol..

[158]  Kang G. Shin,et al.  SWSL: A Synthetic Workload Specification Language for Real-Time Systems , 1994, IEEE Trans. Software Eng..

[159]  Ola Blomkvist,et al.  An Extended Replication of an Experiment for Assessing Methods for Software Requirements Inspections , 1998, Empirical Software Engineering.

[160]  James Miller,et al.  An empirical evaluation of defect detection techniques , 1997, Inf. Softw. Technol..

[161]  Venkataraman Ramesh,et al.  Research in software engineering: an analysis of the literature , 2002, Inf. Softw. Technol..

[162]  Gregory R. Andrews,et al.  The Design of the Saguaro Distributed Operating System , 1987, IEEE Trans. Software Eng..

[163]  Erik H. D'Hollander,et al.  Using hammock graphs to structure programs , 2004, IEEE Transactions on Software Engineering.

[164]  James Miller,et al.  Replicating software engineering experiments: a poisoned chalice or the Holy Grail , 2005, Inf. Softw. Technol..

[165]  David Coppit,et al.  Multiple mass-market applications as components , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[166]  Vianney Côté,et al.  An experiment in software sizing with structured analysis metrics , 1991, J. Syst. Softw..

[167]  A. T. Tai,et al.  On-board maintenance for long-life systems , 1998, Proceedings. 1998 IEEE Workshop on Application-Specific Software Engineering and Technology. ASSET-98 (Cat. No.98EX183).

[168]  C. Wikstrom.  Implementing distributed real-time control systems in a functional programming language , 1996, Proceedings of the 4th International Workshop on Parallel and Distributed Real-Time Systems.

[169]  Walter F. Tichy,et al.  A Controlled Experiment Comparing the Maintainability of Programs Designed with and without Design Patterns—A Replication in a Real Programming Environment , 2004, Empirical Software Engineering.

[170]  P. Triantafiliou.  Independent recovery in large-scale distributed systems , 1996 .

[171]  San-Yih Hwang,et al.  Component and data distribution in a distributed workflow , 1998, Proceedings 1998 Asia Pacific Software Engineering Conference (Cat. No.98EX240).

[172]  Brent Auernheimer,et al.  Visual Depiction of Decision Statements: What is Best for Programmers and Non-Programmers? , 1997, Empirical Software Engineering.

[173]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[174]  Matthias M. Müller,et al.  Two controlled experiments concerning the comparison of pair programming to peer review , 2005, J. Syst. Softw..

[175]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[176]  Des Greer,et al.  Quantitative studies in software release planning under risk and resource constraints , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[177]  Masahito Hirakawa,et al.  Virtual authoring based on the shallow copy technique for a collection of digital documents , 2000, Proceedings International Symposium on Multimedia Software Engineering.

[178]  Dong-Guk Shin,et al.  Fragmenting Relations Horizontally Using a Knowledge-Based Approach , 1991, IEEE Trans. Software Eng..

[179]  Alison Payne.  Designing the databases of the intelligent network , 1992, ICSE 1992.

[180]  Howard J. Hamilton,et al.  A Note on Regeneration with Virtual Copies , 1997, IEEE Trans. Software Eng..

[181]  Chris Kimble,et al.  Uncovering the epistemological and ontological assumptions of software designers , 2004, ArXiv.

[182]  Luigi V. Mancini.  Modular redundancy in a message passing system , 1986, IEEE Transactions on Software Engineering.

[183]  José Javier Dolado,et al.  An investigation of prediction models for project management , 2002, Proceedings 26th Annual International Computer Software and Applications.

[184]  Timothy J. Ellis,et al.  Completing the cycle: meaningful course evaluations , 2003, 33rd Annual Frontiers in Education, 2003. FIE 2003..

[185]  Khaled El Emam,et al.  An Internally Replicated Quasi-Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents , 2001, IEEE Trans. Software Eng..

[186]  Jong-Yoon Kim,et al.  An empirical study of complexity metrics in Cobol programs , 2000, J. Syst. Softw..

[187]  Shari Lawrence Pfleeger,et al.  Experimental design and analysis in software engineering: Part 2: how to set up and experiment , 1995, SOEN.

[188]  Bharat K. Bhargava,et al.  The Raid Distributed Database System , 1989, IEEE Trans. Software Eng..

[189]  Jeffrey C. Carver,et al.  Replicating software engineering experiments: addressing the tacit knowledge problem , 2002, Proceedings International Symposium on Empirical Software Engineering.

[190]  J. Lucas,et al.  Theory-Testing, Generalization, and the Problem of External Validity* , 2003 .

[191]  Peter Triantafillou,et al.  The Location Based Paradigm for Replication: Achieving Efficiency and Availability in Distributed Systems , 1995, IEEE Trans. Software Eng..

[192]  Andrey A. Terekhov.  Re-using software architecture in legacy transformation projects , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[193]  Premkumar T. Devanbu,et al.  Analytical and empirical evaluation of software reuse metrics , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[194]  Benjamin W. Wah,et al.  Learning Dominance Relations in Combinatorial Search Problems , 1988, IEEE Trans. Software Eng..

[195]  Manos Roumeliotis,et al.  A Review of Experimental Investigations into Object-Oriented Technology , 2004, Empirical Software Engineering.