— Mutation testing was initially proposed in the 1970s as intends to guarantee vigour in test case improvement. By making syntactically right substitutions inside the software under test (SUT) and rehashing the test execution stage against the adjusted code, an evaluation could be made of test quality contingent upon if the definitive test cases could locate the code adjustment. Mutation testing is normally used in small code programs, but for a small portion of large program or for a specific code it is used. This paper test a access control part of web based applications for mutant testing, till date testing of web based application is only up to application level only. Generally test cases are executed and stop the testing, but we can not check capability of test cases.

With the rise of software complexity, software-related accidents represent a significant threat for computer-based systems. Software Fault Injection is a method to anticipate worst-case scenarios caused by faulty software through the deliberate injection of software faults. This survey provides a comprehensive overview of the state of the art on Software Fault Injection to support researchers and practitioners in the selection of the approach that best fits their dependability assessment goals, and it discusses how these approaches have evolved to achieve fault representativeness, efficiency, and usability. The survey includes a description of relevant applications of Software Fault Injection in the context of fault-tolerant systems.

When software fault injection is used, faults are typically inserted at the binary or source level. The former is fast but provides poor fault accuracy while the latter cannot scale to large code bases because the program must be rebuilt for each experiment. Alternatives that avoid rebuilding incur large run-time overheads by applying fault injection decisions at run-time. HSFI, our new design, injects faults with all context information from the source level and applies fault injection decisions efficiently on the binary. It places markers in the original code that can be recognized after code generation. We implemented a tool according to the new design and evaluated the time taken per fault injection experiment when using operating systems as targets. We can perform experiments more quickly than other source-based approaches, achieving performance that come close to that of binary-level fault injection while retaining the benefits of source-level fault injection.

Web Services form the basis, not only of standard web based eCommerce applications but also the new Globus implementation of Open GRID Systems Architecture (OGSA). Given the prominence of this technology, test methods and tools are required to ensure that robust, fault tolerant software services are deployed. Testing is required not only to uncover existing problems with the system but to also provide potential users with metrics to compare similar serviced based solutions. Fault injection is a well-proven method of assessing the dependability of a system. Although much work has been done in the area of fault injection and distributed systems in general, there appears to have been little research carried out on applying this to web service based systems [1, 2]. Previous research in the field of service testing via fault injection has concentrated on tightly coupled, RPC based distributed systems [1]. In defining a testing method for web services new sets of challenges are faced which require different solutions. Key differences that are encountered when testing web services are: (1) greater chance for network failure, (2) higher levels of security and encryption, (3) more generic nature of the platform and the need to support multiple programming languages, (4) timing constrains and the asynchronous nature of web service operations. This is due to the loosely coupled nature of typical SOAP based systems that implement services.

We propose a test case selection methodology for object-oriented software. This methodology is based on formal specifications and is aimed at verifying the correctness of method interaction of an object or a cluster of objects. The essence of this methodology is to select test cases by a reduction process. The exhaustive test set, the reference for correctness, is transformed into a practicable test set by applying hypotheses to the objects. We present a set of hypotheses specific to object-oriented systems. Regularity hypotheses are m:n generalizations of the unit behaviour and provides for specifying the shape of test cases. Uniformity hypotheses are 1:n generalizations. Used in conjunction with subdomain decomposition, they guarantee a coverage of the specification. Incrementallity hypotheses take advantage of the subtyping and subclassing relationships to justify the reuse of test cases.

We present the findings of, to the best of our knowledge, the first survey on software testing practices carried out in Australian ICT industry. A total of 65 organizations from various major capital cities in Australia participated in the survey, which was conducted between 2002 and 2003. The survey focused on five major aspects of software testing, namely testing methodologies and techniques, automated testing tools, software testing metrics, testing standards, and software testing training and education. Based on the survey results, current practices in software testing are reported, as well as some observations and recommendations for the future of software testing in Australia for industry and academia.

We argue that higher order mutants are potentially better able to simulate real faults and to reveal insights into bugs than the restricted class of first order mutants. The Mutation Testing community has previously shied away from Higher Order Mutation Testing believing it to be too expensive and therefore impractical. However, this paper argues that Search Based Software Engineering can provide a solution to this apparent problem, citing results from recent work on search based optimization techniques for constructing higher order mutants. We also present a research agenda for the development of Higher Order Mutation Testing.

Various test case selection criteria have been proposed for quality testing of software. It is a common phenomenon that test sets satisfying different criteria have different sizes and fault-detecting ability. Moreover, test sets that satisfy a stronger criterion and detect more faults usually consist of more test cases. A question that often puzzles software testing professionals and researchers is: when a testing criterion C 1 helps to detect more faults than another criterion C 2, is it because C 1 specifically requires test cases that are more fault-sensitive than those for C 2, or is it essentially because C 1 requires more test cases than C 2? In this paper, we discuss several methods and approaches for investigating this question, and empirically compare several common coverage criteria for testing logical decisions, taking into consideration the different sizes of the test sets required by these criteria. Our results demonstrate evidently that the stronger criteria under study are more fault-sensitive than the weaker ones, not solely because the former require more test cases. More importantly, we have illustrated a general approach, which takes into account the size factor of the generated test sets, for demonstrating the superiority of a testing criterion over another.

User Centred Design is accepted as being essential to good software design, only by involving the users throughout the development process can the developers understand what the end users really want (Sharp et al., 2007). One area which has in the past had little experience of User Centred Design is Assistive Technology, and in particular Augmentative and Alternative Communication software (Waller et al., 2005a). Augmentative and Alternative Communication (AAC) offer the possibility of being able to access communication for people with Severe Speech and Physical Impairments. Currently Augmentative and Alternative Communication aids have a high rate of abandonment due in part to poor usability (Prior et al., Accepted). The term abandonment has been used for a variety of different definitions in AAC and assistive technology fields (Johnson et al., 2006). For this thesis, Johnson‟s definition of abandonment will be used; abandonment refers to inappropriate discontinuation of an AAC device. Discontinuation of AAC devices due to a move to a more appropriate device or due to the user no longer having a need for the device are not considered to be an abandonment in this thesis. This abandonment causes frustration and upset to the user, their family and friends and their support staff (Johnson et al., 2006). It has been suggested that by carrying out User Centred Design in the development of Augmentative and Alternative Communication aids that their usability could be improved (Waller et al., 2005a). However, there are many challenges identified in the literature as to how to carry this out with adults with Severe Speech and Physical Impairments (Prior, 2010). 2 This thesis describes a software development study which investigated methods currently used in software development and how they could be adapted for use with this population. The way difficulties cited in the literature when working with this population were tackled are also discussed. The study involved four adults with Severe Speech and Physical Impairments in the User Centred Development of a piece of assistive software. The study found that with careful planning it was possible to conduct User Centred Design with participants with Severe Speech and Physical Impairments, the lessons from this study were translated into recommendations and provided to a second developer who wished to work with adults with Severe Speech and Physical Impairments. The second study found similar levels of contribution to the features of the software were found in the pilot study. This work has demonstrated the potential for adults with Severe Speech and Physical Impairments to be actively involved (i.e. contribute a high proportion of the features) in the development of Augmentative and Alternative Communication software. A number of areas for further investigation have been identified including the differences found in usability of devices developed using User Centred Design compared to traditional methods, and also how adults with Severe Speech and Physical Impairments can be more actively included in a range of research fields.

Though mutation analysis is the primary means of evaluating the quality of test suites, it suffers from inadequate standardization. Mutation analysis tools vary based on language, when mutants are generated (phase of compilation), and target audience. Mutation tools rarely implement the complete set of operators proposed in the literature and mostly implement at least a few domain-specific mutation operators. Thus different tools may not always agree on the mutant kills of a test suite. Few criteria exist to guide a practitioner in choosing the right tool for either evaluating effectiveness of a test suite or for comparing different testing techniques. We investigate an ensemble of measures for evaluating efficacy of mutants produced by different tools. These include the traditional difficulty of detection, strength of minimal sets, and the diversity of mutants, as well as the information carried by the mutants produced. We find that mutation tools rarely agree. The disagreement between scores can be large, and the variation due to characteristics of the project—even after accounting for difference due to test suites—is a significant factor. However, the mean difference between tools is very small, indicating that no single tool consistently skews mutation scores high or low for all projects. These results suggest that experiments yielding small differences in mutation score, especially using a single tool, or a small number of projects may not be reliable. There is a clear need for greater standardization of mutation analysis. We propose one approach for such a standardization.

This work provides an overview of fault injection techniques and their applicability to testing SOAP RPC based Web service systems. We also give a detailed example of the WS-FIT package and use it to detect a problem in a Web service based system.

This short contribution describes first the role of fault injection among the dependability assessment methods that are pertinent approach to the definition and development of dependability benchmarks. Specific problems and challenges faced by dependability benchmarking are then identified and some relevant advances are discussed.

This poster reports on a solution to ERP project cost estimation and on results from its first experimental application.

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way testing is not adequate measured by mutants kill rate compared with hand generated test set of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.

This paper proposes a set of operators for software fault emulation through low-level code mutations. The definition of these operators was based on the analysis of an extensive collection of real software faults. Using the Orthogonal Defect Classification as a starting point, faults were classified in a detailed manner according to the high-level constructs where the faults reside and their effects in the program. We observed that a large percentage of faults fall in well-defined classes and can be characterized in a very precise way, allowing accurate emulation through a small set of mutation operators. The resulting operators closely emulate a broad range of common programmer mistakes. Furthermore, as the mutation is performed directly at the executable code, software faults can be injected in targets for which source code is not available.

This paper proposes a practical way to evaluate the behavior of commercial-off-the-shelf (COTS) operating systems in the presence of faulty device drivers. The proposed method is based on the emulation of software faults in target device drivers and the observation of the behavior of the system and of a workload regarding a comprehensive set of failure modes analyzed according to different dimensions. The emulation of software faults itself is done through the injection at machine-code level of selected mutations that represent the code produced when typical programming errors are made in the high-level language code. An important aspect of the proposed methodology is the use of simple and established practices to evaluate operating systems failure modes, thus allowing its use as a dependability benchmarking technique. The generalization of the methodology to any software system built of discrete and identifiable components is also discussed.

This paper proposes a new technique to emulate software faults by educated mutations introduced at the machine-code level and presents an experimental study on the accuracy of the injected faults. The proposed method consists of finding key programming structures at the machine code-level where high-level software faults can be emulated. The main advantage of emulating software faults at the machine-code level is that software faults can be injected even when the source code of the target application is not available, which is very important for the evaluation of COTS components or for the validation of software fault tolerance techniques in COTS based systems. The technique was evaluated using several real programs and different types of faults and, additionally, it includes our study on the key aspects that may impact on the technique accuracy. The portability of the technique is also addressed. The results show that classes of faults such as assignment, checking, interface, and simple algorithm faults can be directly emulated using this technique.

This paper presents an experimental study on the emulation of software faults by fault injection. In a first experiment, a set of real software faults has been compared with faults injected by a SWIFI tool (Xception) to evaluate the accuracy of the injected faults. Results revealed the limitations of Xception (and other SWIFI tools) in the emulation of different classes of software faults (about 44% of the software faults cannot be emulated). The use of field data about real faults was discussed and software metrics were suggested as an alternative to guide the injection process when field data is nor available. In a second experiment, a set of rules for the injection of errors meant to emulate classes of software faults was evaluated. The fault triggers used seem to be the cause for the observed strong impact of the faults in the target system and in the program results. The results also show the influence in the fault emulation of aspects such as code size, complexity of data structures, and recursive versus sequential execution.

This paper presents an empirical study of the effectiveness of test cases generated from UML state diagrams using transition coverage as the testing criterion. The test cases production is mainly based on an adaptation of a probabilistic method, called statistical testing based on testing criteria. This technique was automated with the aid of the Rational Software Corporation's Rose RealTime tool. The test strategy investigated combines statistical test cases with (few) deterministic test cases focused on domain boundary values. Its feasibility is exemplified on a research version of an avionics system implemented in Java: the Flight Guidance System case study (14 concurrent state diagrams). Then, the results of an empirical evaluation of the effectiveness of the created test cases are presented. The evaluation was performed using mutation analysis to assess the error detection power of the test cases on more than 1500 faults seeded one by one in the Java source code (115 classes, 6500 LOC). A detailed analysis of the test results allows us to draw first conclusions on the expected strengths and weaknesses of the proposed test strategy.

This paper presents an empirical study of an important aspect of software defect behavior: the propagation of data-state errors. A data-state error occurs when a fault is executed and affects a program's data-state, and it is said to propagate if it affects the outcome of the execution. Our results show that data-state errors appear to have a property that is quite useful when simulating faulty code: for a given input, it appears that either all data state errors injected at a given location tend to propagate to the output, or else none of them do. These results are interesting because of what they indicate about the behavior of data-state errors in software. They suggest that data state errors behave in an orderly way, and that the behavior of software may not be as unpredictable as it could theoretically be. Additionally, if all faults behave the same for a given input and a given location, then one can use simulation to get a good picture of how faults behave, regardless of whether the faults one has simulated are representative of real faults.