A Decision Tree Based Methodology for Evaluating Creativity in Engineering Design

Multiple metrics have been proposed to measure the creativity of products, yet there is still a need for effective, reliable methods to assess the originality of new product designs. In the present article we introduce a method to assess the originality of concepts that are produced during idea generation activities within engineering design. This originality scoring method uses a decision tree that is centered around distinguishing design innovations at the system level. We describe the history and the development of our originality scoring method, and provide evidence of its reliability and validity. A full protocol is provided, including training procedures for coders and multiple examples of coded concepts that received different originality scores. We summarize data from over 500 concepts for garbage collection systems that were scored by Kershaw et al. (2015). We then show how the originality scoring method can be applied to a different design problem. Our originality scoring method, the Decision Tree for Originality Assessment in Design (DTOAD), has been a useful tool to identify differences in originality between various cohorts of Mechanical Engineering students. The DTOAD reveals cross-sectional differences in creativity between beginning and advanced students, and shows longitudinal growth in creativity from the beginning to the end of the undergraduate career, thus showing how creativity can be influenced by the curriculum. The DTOAD can be applied to concepts produced using different ideation procedures, including concepts produced both with and without a baseline example product, and concepts produced when individuals are primed to think of different users for their designs. Finally, we show how our the DTOAD compares to other measurements of creativity, such as novelty, fixation, and remoteness of association.

[1]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[2]  L. Allbon,et al.  Creative Problem Solving , 1968, Canadian journal of occupational therapy. Revue canadienne d'ergotherapie.

[3]  Luis A. Vasconcelos,et al.  Inspiration and fixation: Questions, methods, findings, and challenges , 2016 .

[4]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[5]  E. Kemps,et al.  General lifestyle activities as a predictor of current cognition and cognitive change in older adults: a cross-sectional and longitudinal examination. , 2005, The journals of gerontology. Series B, Psychological sciences and social sciences.

[6]  J. W. Getzels,et al.  The Creative Vision: A Longitudinal Study of Problem Finding in Art , 1976 .

[7]  Steven M. Smith,et al.  Incubation and the persistence of fixation in problem solving. , 1991, The American journal of psychology.

[8]  L.J. Leifer,et al.  Engineering design thinking, teaching, and learning , 2005, IEEE Engineering Management Review.

[9]  Michael Scaife,et al.  Cognitive Science Approaches To Understanding Diagrammatic Representations , 2001, Artificial Intelligence Review.

[10]  Amaresh Chakrabarti,et al.  Assessing design creativity , 2011 .

[11]  Carolyn Conner Seepersad,et al.  An Experimental Investigation of the Effectiveness of Empathic Experience Design for Innovative Concept Generation , 2014 .

[12]  Michelene T. H. Chi,et al.  Observing Tutorial Dialogues Collaboratively: Insights About Human Tutoring Effectiveness From Vicarious Learning , 2008, Cogn. Sci..

[13]  Trina C. Kershaw,et al.  The Relationship Between Fixation and Originality in Undergraduate Mechanical Engineering Students , 2017 .

[14]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[15]  Shanna R. Daly,et al.  Can experienced designers learn from new tools? A case study of idea generation in a professional engineering team , 2014 .

[16]  S. Dollinger,et al.  Note on Consensual Assessment Technique in Creativity Research , 2005, Perceptual and motor skills.

[17]  Mary A. Keetz.  An Experimental Investigation of the Effectiveness of A College Reading and Study Skills Course for Freshmen Students Enrolled in Scientific Courses of Study. , 1970 .

[18]  Rebecca Peterson,et al.  The Influence of Group Interaction on Creativity in Engineering Design , 2016, CogSci.

[19]  Giovanni Emanuele Corazza,et al.  Potential Originality and Effectiveness: The Dynamic Definition of Creativity , 2016 .

[20]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[21]  Carolyn Conner Seepersad,et al.  A cross-sectional and longitudinal examination of the development of innovation capability in undergraduate engineering students , 2015 .

[22]  T. Lubart.  The 7 C's of Creativity. , 2017 .

[23]  David H. Cropley,et al.  The Cambridge Handbook of Creativity: Functional Creativity , 2010 .

[24]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[25]  Hershey H. Friedman,et al.  Rating the Rating Scales , 1999 .

[26]  Christine Charyton.  Creative Engineering Design Assessment , 2014 .

[27]  Susan R. Goldman,et al.  Knowledge of connectors as cohesion devices in text: a comparative study of native-english and english-as-a-second-language speakers , 1992 .

[28]  Carolyn Conner Seepersad,et al.  Crowd-sourcing the evaluation of creativity in conceptual design: A pilot study , 2014 .

[29]  Susan R. Goldman,et al.  Complex Mathematical Problem Solving by Individuals and Dyads , 1997 .

[30]  Carolyn Conner Seepersad,et al.  The effects of the undergraduate curriculum and individual differences on student innovation capabilities , 2014 .

[31]  S. Rostan.  Studio Learning: Motivation, Competence, and the Development of Young Art Students' Talent and Creativity , 2010 .

[32]  S Tippett.  Student outcomes. , 1999, Physical therapy.

[33]  David C. Brown.  Problems with the Calculation of Novelty Metrics , 2014 .

[34]  Carolyn Conner Seepersad,et al.  A Study of the Effectiveness of Empathic Experience Design as a Creativity Technique , 2011 .

[35]  M. Runco,et al.  Theories of Creativity , 1990, Encyclopedia of Creativity.

[36]  Maria C. Yang,et al.  A Step Beyond to Overcome Design Fixation: A Design-by-Analogy Approach , 2015 .

[37]  T. M. Amabile.  Social psychology of creativity: A consensual assessment technique. , 1982 .

[38]  Jason L. G. Braasch,et al.  The influences of text and reader characteristics on learning from refutations in science texts. , 2013 .

[39]  Laroche Edouard,et al.  Cardialockプロジェクト:心臓手術用の能動型安定化装置設計 | 文献情報 | J-GLOBAL 科学技術総合リンクセンター , 2011 .

[40]  R. Sternberg,et al.  The concept of creativity: Prospects and paradigms. , 1998 .

[41]  A. Cantor,et al.  Sample-size calculations for Cohen's kappa. , 1996 .

[42]  J. Duderstadt.  Engineering for a Changing World , 2010 .

[43]  Scarlett R. Miller,et al.  The Impact of Example Modality and Physical Interactions on Design Creativity , 2014 .

[44]  Jonathan Evans,et al.  Problem-solving Strategies and Expertise in Engineering Design. , 1997 .

[45]  John Baer,et al.  Creativity in the Engineering Domain , 2017 .

[46]  Trina C. Kershaw,et al.  The Influence of Physical Examples on Originality and Fixation in Engineering Design , 2018 .

[47]  Richard W. Hass,et al.  Revisiting the 10-year rule for composers from the Great American Songbook: On the validity of two measures of creative production. , 2015 .

[48]  David H. Cropley,et al.  Furious activity vs. understanding: How much expertise is needed to evaluate creative work? , 2013 .

[49]  Saskia Jaarsveld,et al.  Intelligence and Creativity in Problem Solving: The Importance of Test Features in Cognition Research , 2017, Front. Psychol..

[50]  D. Simonton.  Creativity and Discovery as Blind Variation: Campbell's (1960) BVSR Model after the Half-Century Mark , 2011 .

[51]  Matthijs Baas,et al.  The dual pathway to creativity model: Creative ideation as a function of flexibility and persistence , 2010 .

[52]  K. Dunbar.  How scientists think: On-line creativity and conceptual change in science. , 1997 .

[53]  T. Lubart,et al.  Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges , 2014 .

[54]  Karl K. Jeffries,et al.  A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity? , 2017 .

[55]  Jonathan Cagan,et al.  On the benefits and pitfalls of analogies for innovative design : Ideation performance based on analogical distance, commonness, and modality of examples , 2011 .

[56]  Mark Batey,et al.  The Measurement of Creativity: From Definitional Consensus to the Introduction of a New Heuristic Framework , 2009 .

[57]  Paul Goodwin,et al.  Decision Analysis for Management Judgment , 1998 .

[58]  John Baer,et al.  The Cambridge Handbook of Creativity Across Domains , 2017 .

[59]  Mark A. Runco,et al.  Everyone has creative potential. , 2004 .

[60]  Carolyn Conner Seepersad,et al.  Study of Existing Metrics Used in Measurement of Ideation Effectiveness , 2010 .

[61]  Trina C. Kershaw,et al.  Practice makes proficient: teaching undergraduate students to understand published research , 2018 .

[62]  N. Cross,et al.  Design Expertise Amongst Student Designers , 1994 .

[63]  David H. Cropley.  The Importance of Creativity in Engineering , 2015 .

[64]  Matthew C. Makel,et al.  Assessment of Creativity , 2010, Creativity.

[65]  John Merrill,et al.  CEDA: A research instrument for creative engineering design assessment. , 2008 .

[66]  J. Guilford.  The structure of intellect. , 1956, Psychological bulletin.

[67]  James C. Kaufman,et al.  Assessing Creativity Using the Consensual Assessment Technique , 2009 .

[68]  S. Mednick.  The associative basis of the creative process. , 1962, Psychological review.

[69]  A. Cropley.  In Praise of Convergent Thinking , 2006 .

[70]  Dor Abrahamson,et al.  Reinventing discovery learning: a field-wide research program , 2018 .

[71]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[72]  Robert J. Youmans,et al.  The effects of physical prototyping and group work on the reduction of design fixation , 2011 .

[73]  David H. Cropley,et al.  Measuring Functional Creativity: Non‐Expert Raters and the Creative Solution Diagnosis Scale , 2012 .

[74]  P. Silvia,et al.  A First Look at the Role of Domain-General Cognitive and Creative Abilities in Jazz Improvisation , 2013 .

[75]  Robert J. Sternberg,et al.  The Creativity Conundrum: A Propulsion Model of Kinds of Creative Contributions , 2001 .

[76]  C. Bao Iturbe,et al.  Educating the engineer of 2020: Adapting engineering education to the new century , 2009, IEEE Engineering Management Review.

[77]  Lior Fink,et al.  Balancing volume and duration of information consumption by physicians: The case of health information exchange in critical care , 2017, J. Biomed. Informatics.

[78]  Irem Y. Tumer,et al.  A comparison of creativity and innovation metrics and sample validation through in-class design projects , 2013 .

[79]  Na Li,et al.  Translating the ICAP Theory of Cognitive Engagement Into Practice , 2018, Cogn. Sci..

[80]  Mareike B. Wieth,et al.  Conflicts and Consistencies in Creativity Research and Teaching , 2018, Teaching of Psychology.

[81]  Kevin Otto,et al.  Product Design: Techniques in Reverse Engineering and New Product Development , 2000 .

[82]  John T. Willse,et al.  Assessing Creativity With Divergent Thinking Tasks: Exploring the Reliability and Validity of New Subjective Scoring Methods , 2008 .

[83]  D. Campbell.  Blind variation and selective retention in creative thought as in other knowledge processes. , 1960, Psychological review.

[84]  Cynthia J. Atman,et al.  A comparison of freshman and senior engineering design processes , 1999 .

[85]  M. Chi.  Quantifying Qualitative Analyses of Verbal Data: A Practical Guide , 1997 .