5.4.1 An Experimental Methodology to Evaluate Concept Generation Procedures Based on Quantitative Lifecycle Performance

This study presents an experimental methodology to measure how concept generation procedures can affect the anticipated lifecycle performance of engineering systems design concepts. The methodology is based on objective and quantitative measurements of anticipated lifecycle performance of the design concepts. It merges cognitive and computer‐aided techniques from the fields of collaboration engineering, creativity, and engineering design. It complements the body of existing techniques relying on subjective expert assessments, and other objective metrics not explicitly measuring anticipated lifecycle performance (e.g. development time and cost). Application of the methodology is demonstrated through evaluation of design procedures generating flexibility in engineering systems design. Experiments had ninety participants generate creative design alternatives to a simplified real estate development design problem. Thirty‐two teams of two to three participants performed the collaborative design exercise. An online Group‐Support System interface enabled efficient data collection and analysis. A computationally efficient mid‐fidelity model was used to evaluate flexible design concepts quantitatively based on real options analysis techniques.

[1]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[2]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[3]  E. Pitman.  SIGNIFICANCE TESTS WHICH MAY BE APPLIED TO SAMPLES FROM ANY POPULATIONS III. THE ANALYSIS OF VARIANCE TEST , 1938 .

[4]  A. Osborn.  Applied Imagination: Principles and Procedures of Creative Thinking , 1953 .

[5]  R. W. Morgan,et al.  How to design an experiment. , 1966, Annals of physical medicine.

[6]  E. Prescott,et al.  Investment Under Uncertainty , 1971 .

[7]  F. Black,et al.  The Pricing of Options and Corporate Liabilities , 1973, Journal of Political Economy.

[8]  S. Myers.  Determinants of corporate borrowing , 1977 .

[9]  S. Ross,et al.  Option pricing: A simplified approach☆ , 1979 .

[10]  田口 玄一,et al.  System of experimental design : engineering methods to optimize quality and minimize costs , 1987 .

[11]  William J. Welch,et al.  Construction of Permutation Tests , 1990 .

[12]  David Craig,et al.  The importance of drawing in the mechanical design process , 1990, Comput. Graph..

[13]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1993 .

[14]  J. Dixon,et al.  Engineering Design , 2019, Springer Handbook of Mechanical Engineering.

[15]  E. Salas,et al.  Productivity loss in brainstorming groups: A meta-analytic integration. , 1991 .

[16]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[17]  Durward K. Sobek,et al.  The Second Toyota Paradox: How Delaying Decisions Can Make Better Cars Faster , 1995 .

[18]  Jay F. Nunamaker,et al.  Lessons from a Dozen Years of Group Support Systems Research: A Discussion of Lab and Field Findings , 1996, J. Manag. Inf. Syst..

[19]  R. Audi.  Epistemology: A Contemporary Introduction to the Theory of Knowledge , 1997 .

[20]  George A. Hazelrigg,et al.  A Framework for Decision-Based Engineering Design , 1998 .

[21]  Gabriela Goldschmidt,et al.  Contents and Structure in Design Reasoning , 1998 .

[22]  S. Shreve,et al.  Stochastic differential equations , 1955, Mathematical Proceedings of the Cambridge Philosophical Society.

[23]  Nalin Kulatilaka,et al.  Real Options: Managing Strategic Investment in an Uncertain World , 1998 .

[24]  Robert P. Bostrom,et al.  Research in creativity and GSS , 1998, Proceedings of the Thirty-First Hawaii International Conference on System Sciences.

[25]  杨文秀,et al.  此处“personality”译法探析 , 2000 .

[26]  C. F. Jeff Wu,et al.  Experiments: Planning, Analysis, and Parameter Design Optimization , 2000 .

[27]  L. Christensen,et al.  Educational Research: Quantitative and Qualitative Approaches , 2000 .

[28]  Jami J. Shah,et al.  Evaluation of idea generation methods for conceptual design: Effectiveness metrics and design of experiments , 2000 .

[29]  Eileen M. Trauth,et al.  Understanding Computer-Mediated Discussions: Positivist and Interpretive Analyses of Group Support System Use , 2000, MIS Q..

[30]  Warren E. Walker,et al.  Adaptive policies, policy analysis, and policy-making , 2001, Eur. J. Oper. Res..

[31]  T. Copeland.  Real Options: A Practitioner's Guide , 2001 .

[32]  Jesse C Arnold Professor Emeritus,et al.  Introduction to Probability and Statistics: Principles and Applications for Engineering and the Computing Sciences , 1990 .

[33]  Bernd Bender,et al.  Re-Interpretation of Conceptualisation — A Contribution to the Advance of Design Theory , 2003 .

[34]  Udo Lindemann.  Human behaviour in design : individuals, teams, tools , 2003 .

[35]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[36]  Douglas R. Vogel,et al.  Fifteen years of GSS in the field: a comparison across time and national boundaries , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[37]  Udo Lindemann.  Human Behaviour in Design , 2003 .

[38]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[39]  Robert O. Briggs,et al.  Causal Relationships in Creative Problem Solving: Comparing Facilitation Interventions for Ideation , 2004, J. Manag. Inf. Syst..

[40]  Leyuan Shi,et al.  A real-options-based analysis for supply chain decisions , 2005 .

[41]  Kristin L. Wood,et al.  Functional Representations in Conceptual Design: A First Study in Experimental Design and Evaluation , 2005 .

[42]  Eamonn O'Neill,et al.  Understanding design as a social creative process , 2005, C&C '05.

[43]  Armin P. Schulz,et al.  Design for changeability (DfC): Principles to enable changes in systems throughout their entire lifecycle , 2005, Syst. Eng..

[44]  Nikolaos Georgiopoulos.  Real Options , 2006 .

[45]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[46]  Eun Suk Suh,et al.  Flexible product platforms: framework and case study , 2007, DAC 2006.

[47]  D. Braha,et al.  Complex Engineered Systems: A New Paradigm , 2006 .

[48]  R. Neufville,et al.  Design of Engineering Systems under Uncertainty via Real Options and Heuristic Optimization , 2006 .

[49]  G. Rothwell.  A real options approach to-evaluating new nuclear power plants , 2006 .

[50]  Daniel D. Frey,et al.  Validation of design methods: lessons from medicine , 2006 .

[51]  R. Briggs,et al.  Meeting Satisfaction for Technology-Supported Groups , 2006 .

[52]  Xiang Li,et al.  Model-Based Validation of Design Methods , 2006 .

[53]  Jay F. Nunamaker,et al.  On the Measurement of Ideation Quality , 2007, J. Manag. Inf. Syst..

[54]  Maria C. Yang,et al.  Design Methods, Tools, and Outcome Measures: A Survey of Practitioners , 2007 .

[55]  Christopher J. Jablonowski,et al.  Estimating the Cost of Errors in Estimates Used During Concept Selection , 2008 .

[56]  Olivier L. de Weck,et al.  Exploring Flexible Strategies in Engineering Systems Using Screening Models Applications to Offshore Petroleum Projects by Jijun Lin , 2008 .

[57]  Claudia Eckert,et al.  Engineering change: drivers, sources and approaches in industry , 2009 .

[58]  Matthew I. Campbell,et al.  An experimental study on the effects of a computational design tool on concept generation , 2009 .

[59]  Richard de Neufville,et al.  Vertical phasing as a corporate real estate strategy and development option , 2009 .

[60]  Fumihiko Kimura,et al.  Design methodologies: Industrial and educational applications , 2009 .

[61]  David W. Rosen,et al.  Refined metrics for measuring ideation effectiveness , 2009 .

[62]  Robert O. Briggs,et al.  Practitioners vs Facilitators a Comparison of Participant Perceptions on Success , 2009, 2009 42nd Hawaii International Conference on System Sciences.

[63]  Michel-Alexandre Cardin,et al.  Integrated Method for Designing Valuable Flexibility In Oil Development Projects , 2009 .

[64]  Joshua D. Summers,et al.  Introduction of Design Enabling Tools , 2009 .

[65]  Steve Caplin,et al.  Principles Of Design , 2011 .

[66]  Maria C. Yang,et al.  Observations on concept generation and sketching in engineering design , 2009 .

[67]  Yingxia Yang,et al.  A screening model to explore planning decisions in automotive manufacturing systems under demand uncertainty , 2009 .

[68]  Y. Reich.  My method is better! , 2010 .

[69]  Alexander Verbraeck,et al.  A DEVS component library for simulation-based design of automated container terminals , 2010, SimuTools.

[70]  Jonathan Cagan,et al.  A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty , 2010 .

[71]  Daniel E. Hastings,et al.  Real Options in Enterprise Architecture: A Holistic Mapping of Mechanisms and Types for Uncertainty Management , 2011, IEEE Transactions on Engineering Management.

[72]  R de Neufville,et al.  Flexibility in engineering design (engineering systems) , 2011 .

[73]  Michel-Alexandre Cardin,et al.  Quantitative performance-based evaluation of a procedure for flexible design concept generation , 2011 .

[74]  Vladan Babovic,et al.  A Real Options Approach to the Design and Architecture of Water Supply Systems Using Innovative Water Technologies Under Uncertainty , 2012 .

[75]  Daniel D. Frey,et al.  Empirical evaluation of procedures to generate flexibility in engineering systems and improve lifecycle performance , 2013 .