A Study of Technical Engineering Peer Reviews at NASA

This report describes the state of practices of design reviews at NASA and research into what can be done to improve peer review practices. There are many types of reviews at NASA: required and not, formalized and informal, programmatic and technical. Standing project formal reviews such as the Preliminary Design Review and Critical Design Review are a required part of every project and mission development. However, the technical, engineering peer reviews that support teams' work on such projects are informal, some times ad hoc, and inconsistent across the organization. The goal of this work is to identify best practices and lessons learned from NASA's experience, supported by academic research and methodologies to ultimately improve the process. This research has determined that the organization, composition, scope, and approach of the reviews impact their success. Failure Modes and Effects Analysis (FMEA) can identify key areas of concern before or in the reviews. Product definition tools like the Project Priority Matrix, engineering-focused Customer Value Chain Analysis (CVCA), and project or system-based Quality Function Deployment (QFD) help prioritize resources in reviews. The use of information technology and structured design methodologies can strengthen the engineering peer review process to help NASA work towards error-proofing the design process.

[1]  Kosuke Ishii,et al.  Design Process Error-Proofing: Development of Automated Error-Proofing Information Systems , 2003, DAC 2003.

[2]  James R. Wertz,et al.  Space Mission Analysis and Design , 1992 .

[3]  Françoise Détienne,et al.  Quantitative Measurements of the Influence of Participant Roles during Peer Review Meetings , 2001, Empirical Software Engineering.

[4]  Elaine J. Weyuker.  Predicting project risk from architecture reviews , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[5]  Lesley Trenner.  Prototyping for usability: the benefits of peer group review , 1995 .

[6]  J. O'Reilly.  Risk, adventure and the tyranny of peer review , 2002 .

[7]  J. R. Friedman.  An experiment in processing electronic documents , 1997, Proceedings of IPCC 97. Communication.

[8]  Nancy G. Leveson.  Intent Specifications: An Approach to Building Human-Centered Specifications , 2000, IEEE Trans. Software Eng..

[9]  D. Dörner.  The logic of failure. , 1990, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[10]  P. Eibeck.  Criteria for peer-review of engineering courseware on the NEEDS database , 1996 .

[11]  Kosuke Ishii,et al.  Design Process Error Proofing: Failure Modes and Effects Analysis of the Design Process , 2007 .

[12]  Edward F. Gehringer,et al.  Strategies and mechanisms for electronic peer review , 2000, 30th Annual Frontiers in Education Conference. Building on A Century of Progress in Engineering Education. Conference Proceedings (IEEE Cat. No.00CH37135).