Robust Testing and Evaluation of Systems: Framework, Approaches, and Illustrative Tools

Testing and evaluation (T&E) will be successful if its plans, methods, and analysis components perform adequately despite departures from ideal conditions. A framework of robust T&E approaches is presented that is aimed at avoiding or gracefully accommodating such departures (e.g., resource losses and schedule changes). Broadening this framework, illustrative tools and other guidance are described for building robust T&E plans, methods, and analysis components. The T&E practitioner can use this broadened framework of robust approaches, tools, and guidance to ensure acceptable results despite departures from the ideal.

[1]  H. Huynh,et al.  Estimation of the Box Correction for Degrees of Freedom from Sample Data in Randomized Block and Split-Plot Designs , 1976 .

[2]  E C Poulton,et al.  Unwanted asymmetrical transfer effects with balanced experimental designs. , 1966, Psychological bulletin.

[3]  Alvah C. Bittner,et al.  Human Factors Measurement: Nature, Problems, and Strengthening , 1990 .

[4]  Marvin C. McCallum,et al.  Empirical Identification of User Information Requirements in Command and Control Evaluation , 1990 .

[5]  F B Baker,et al.  Estimates of test size for several test procedures based on conventional variance ratios in the repeated measures design , 1967, Psychometrika.

[6]  Alvah C. Bittner Robust Testing and Evaluation (T&E) of Systems , 1991 .

[7]  John C. Guignard,et al.  HUMAN FACTORS ENGINEERING PRINCIPLES FOR MINIMIZING ADVERSE SHIP MOTION EFFECTS: THEORY AND PRACTICE , 1985 .

[8]  David Meister,et al.  Behavioral Analysis and Measurement Methods , 1985 .

[9]  Barry H. Kantowitz Can Cognitive Theory Guide Human Factors Measurement? , 1990 .

[10]  R. C. Carter,et al.  Performance Evaluation Tests for Environmental Research (Peter): Evaluation of 114 Measures , 1984, Perceptual and motor skills.

[11]  D. R. Towill,et al.  Transfer Functions and Learning Curves , 1976 .

[12]  Susan G. Hill,et al.  Workload Assessment of a Remotely Piloted Vehicle (RPV) System , 1988 .

[13]  R. F. Brown,et al.  PERFORMANCE EVALUATION , 2019, ISO 22301:2019 and business continuity management – Understand how to plan, implement and enhance a business continuity management system (BCMS).

[14]  D. Damos,et al.  Individual Differences in Multiple-Task Performance as a Function of Response Strategy , 1983, Human factors.

[15]  J. C. Guignard,et al.  Shipboard Evaluation of Motion Sickness Incidence and Human Engineering Problems , 1988 .

[16]  A. O. Dick,et al.  Operator Workload: Comprehensive Review and Evaluation of Operator Workload Methodologies , 1989 .

[17]  J. C. Byers,et al.  Comparison of Four Subjective Workload Rating Scales , 1992 .

[18]  B H Kantowitz,et al.  Selecting Measures for Human Factors Research , 1992, Human factors.

[19]  Alvah C. Bittner,et al.  Repeated-measures analysis: Issues and options , 1992 .