The Relationship Between Design Outcomes and Mental States During Ideation

Using electroencephalography (EEG) to predict design outcomes could be used in many applications as it facilitates the correlation of engagement and cognitive workload with ideation effectiveness. It also establishes a basis for the connection between EEG measurements and common constructs in engineering design research. In this paper, we propose a support vector machine (SVM)-based prediction model for design outcomes using EEG metrics and some demographic factors as predictors. We trained and validated the model with more than 100 concepts, and then evaluated the relationship between EEG data and concept-level measures of novelty, quality, and elaboration. The results characterize the combination of engagement and workload that is correlated with good design outcomes. Findings also suggest that EEG technologies can be used to partially replace or augment traditional ideation metrics and to improve the efficacy of ideation research. [DOI: 10.1115/1.4036131]

[1]  Carole R. Beal,et al.  EEG estimates of engagement and cognitive workload predict math problem solving outcomes , 2012, UMAP.

[2]  Arun Khosla,et al.  Comparison of SVM and ANN for classification of eye events in EEG , 2011 .

[3]  John S. Gero,et al.  An approach to the analysis of design protocols , 1998 .

[4]  C. McBride-Chang,et al.  Age Differences in Creativity: Task Structure and Knowledge Base , 2005 .

[5]  Carolyn Conner Seepersad,et al.  Study of Existing Metrics Used in Measurement of Ideation Effectiveness , 2010 .

[6]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[7]  Jonathan Cagan,et al.  A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty , 2010 .

[8]  J. Guilford,et al.  The nature of human intelligence. , 1968 .

[9]  K. Miller.  Executive functions. , 2005, Pediatric annals.

[10]  Irem Y. Tumer,et al.  A comparison of creativity and innovation metrics and sample validation through in-class design projects , 2013 .

[11]  Barry Kudrowitz,et al.  The Effects of Elaboration in Creativity Tests as it Pertains to Overall Scores and How it Might Prevent a Person From Thinking of Creative Ideas During the Early Stages of Brainstorming and Idea Generation , 2015 .

[12]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[13]  John R. Anderson.  Retrieval of propositional information from long-term memory , 1974 .

[14]  Steven E. Stemler.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation A Comparison of Consensus, Consistency, and Measurement A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability Approaches to Estimating Interrater Reliabilit , 2022 .

[15]  Ellen Yi-Luen Do,et al.  Extended linkography and distance graph in design evaluation: an empirical study of the dual effects of inspiration sources in creative design , 2010 .

[16]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[17]  Yong Zeng,et al.  CLUSTERING DESIGNERS' MENTAL ACTIVITIES BASED ON EEG POWER , 2012 .

[18]  Susan P. Besemer.  Creative Product Analysis Matrix: Testing the Model Structure and a Comparison Among Products--Three Novel Chairs , 1998 .

[19]  Alice M. Agogino,et al.  Automatically Inferring Metrics for Design Creativity , 2013 .

[20]  Teresa M. Amabile,et al.  Assessing the Work Environment for Creativity , 1996 .

[21]  Hui Xiong,et al.  Enhancing data analysis with noise removal , 2006, IEEE Transactions on Knowledge and Data Engineering.

[22]  Jami J. Shah,et al.  Empirical Studies of Designer Thinking: Past, Present, and Future , 2015 .

[23]  Teresa M. Amabile,et al.  Componential Theory of Creativity , 2012 .

[24]  Yan Jin,et al.  Impact of Mental Iteration on Concept Generation , 2006 .

[25]  Jonathan Stephen Fish,et al.  Amplifying the Mind’s Eye: Sketching and Visual Cognition , 1990 .

[26]  Diana G. Oblinger,et al.  Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge. A 21st Century Agenda for the National Science Foundation , 2008 .

[27]  Carole R. Beal,et al.  EEG Estimates of Cognitive Workload and Engagement Predict Math Problem Solving Outcomes. , 2012 .

[28]  G. Goldschmidt.  The dialectics of sketching , 1991 .

[29]  Karen O'Quin,et al.  Using the Creative Product Semantic Scale as a Metric for Results-Oriented Business , 2006 .

[30]  Andrian Marcus,et al.  Data Cleansing: Beyond Integrity Analysis , 2000, IQ.

[31]  Tomás Dorta,et al.  The ideation gap:: hybrid tools, design flow and practice , 2008 .

[32]  Timothy W. Simpson,et al.  The impact of virtual product dissection environments on student design learning and self-efficacy , 2015 .

[33]  Thanh An Nguyen,et al.  ANALYSIS OF DESIGN ACTIVITIES USING EEG SIGNALS , 2010 .

[34]  Yong Zeng,et al.  A physiological study of relationship between designer's mental effort and mental stress during conceptual design , 2014, Comput. Aided Des..

[35]  Jon Touryan,et al.  A Comparison of Electroencephalography Signals Acquired from Conventional and Mobile Systems , 2014 .

[36]  Jay F. Nunamaker,et al.  On the Measurement of Ideation Quality , 2007, J. Manag. Inf. Syst..

[37]  Wilmar B. Schaufeli,et al.  Defining and measuring work engagement : Bringing clarity to the concept , 2010 .

[38]  M. Runco,et al.  Cognition and creativity , 1995 .

[39]  Erin A. Carroll,et al.  Quantifying the personal creative experience: evaluation of digital creativity support tools using self-report and physiological responses , 2013 .

[40]  Jonathan Cagan,et al.  Understanding Consumer Tradeoffs Between Form and Function Through Metaconjoint and Cognitive Neuroscience Analyses , 2013 .

[41]  Julie S. Linsey,et al.  In Search of Effective Design Problems for Design Research , 2015 .

[42]  Chris Berka,et al.  Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model , 2011, Biological Psychology.

[43]  Robert W. Proctor,et al.  Human factors in simple and complex systems , 1993 .

[44]  D. Hocevar,et al.  The Development of the Creative Behavior Inventory (CBI). , 1979 .

[45]  Ryad Titah,et al.  Measure for Measure: A two study multi-trait multi-method investigation of construct validity in IS research , 2013, Comput. Hum. Behav..

[46]  Joran W. Booth,et al.  Reducing Sketch Inhibition During Concept Generation: Psychophysiological Evidence of the Effect of Interventions , 2015 .

[47]  Wenji Mao,et al.  Cyber-Physical-Social Systems for Command and Control , 2011, IEEE Intelligent Systems.

[48]  S. Gilbert,et al.  Exploring the neurological basis of design cognition using brain imaging: some preliminary results , 2009 .

[49]  Marti J. Anderson,et al.  Permutation tests for univariate or multivariate analysis of variance and regression , 2001 .

[50]  C. Martindale.  Biological bases of creativity. , 1998 .

[51]  John Merrill,et al.  CEDA: A research instrument for creative engineering design assessment. , 2008 .

[52]  Andrian Marcus,et al.  Data Cleansing: Beyond Integrity Analysis 1 , 2000 .

[53]  Yan Jin,et al.  Cognitive Modeling of Iteration in Conceptual Design , 2004 .

[54]  O. Wilhelm,et al.  Working memory capacity - facets of a cognitive ability construct , 2000 .

[55]  Karthik Ramani,et al.  Interventions for teaching sketching skills and reducing inhibition for novice engineering designers , 2016 .

[56]  G. Borghini,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[57]  Michelle N. Lumicao,et al.  EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. , 2007, Aviation, space, and environmental medicine.

[58]  Irem Y. Tumer,et al.  The Potential of Creativity Metrics for Mechanical Engineering Concept Design , 2009 .

[59]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[60]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[61]  Durward K. Sobek,et al.  Relating Design Process to Quality: A Virtual Design of Experiments Approach , 2007 .

[62]  Celine Latulipe,et al.  Triangulating the personal creative experience: self-report, external judgments, and physiology , 2012, Graphics Interface.

[63]  David Wallace,et al.  Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation , 2010 .

[64]  Sally M. Reis,et al.  The Assessment of Creative Products in Programs for Gifted and Talented Students , 1991 .

[65]  Claude Frasson,et al.  Classification and Regression of Learner’s Scores in Logic Environment , 2015 .

[66]  Glenn F. Wilson,et al.  Psychophysiological responses to changes in workload during simulated air traffic control , 1996, Biological Psychology.

[67]  Vimal Viswanathan,et al.  A study on the role of physical models in the mitigation of design fixation , 2014 .

[68]  D. Russell,et al.  Where the Art is: Defining and Managing Creativity in New Media Smes , 2002 .

[69]  Edward M. Bowden,et al.  The origins of insight in resting-state brain activity , 2008, Neuropsychologia.

[70]  Anthony J. Ries,et al.  Usability of four commercially-oriented EEG systems , 2014, Journal of neural engineering.

[71]  M. Runco.  Children's divergent thinking and creative ideation , 1992 .

[72]  James M. LeBreton,et al.  Answers to 20 Questions About Interrater Reliability and Interrater Agreement , 2008 .

[73]  David W. Rosen,et al.  Refined metrics for measuring ideation effectiveness , 2009 .

[74]  D. Simonton.  Creative thought as blind-variation and selective-retention: combinatorial models of exceptional creativity. , 2010, Physics of life reviews.

[75]  S. G. Hart,et al.  Development of NASA-TLX(Task Load Index) , 1988 .

[76]  Erin A. Carroll.  Convergence of self-report and physiological responses for evaluating creativity support tools , 2011, C&C '11.

[77]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[78]  Douglas L. Dean,et al.  Identifying Good Ideas: Constructs and Scales for Idea Evaluation , 2006 .

[79]  Chris Berka,et al.  Real-Time Analysis of EEG Indexes of Alertness, Cognition, and Memory Acquired With a Wireless EEG Headset , 2004, Int. J. Hum. Comput. Interact..

[80]  Marine Agogué,et al.  The impact of age and training on creativity: A design-theory approach to study fixation effects , 2014 .

[81]  T. M. Amabile.  Social psychology of creativity: A consensual assessment technique. , 1982 .

[82]  T. Chau,et al.  Effects of user mental state on EEG-BCI performance , 2015, Front. Hum. Neurosci..

[83]  Johan Hagelbäck,et al.  Evaluating Classifiers for Emotion Recognition Using EEG , 2013, HCI.

[84]  John S. Gero,et al.  Design and other types of fixation , 1996 .

[85]  Xiwen Sun,et al.  Age-dependent brain activation during forward and backward digit recall revealed by fMRI , 2005, NeuroImage.

[86]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[87]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[88]  Carolyn Conner Seepersad,et al.  A cross-sectional and longitudinal examination of the development of innovation capability in undergraduate engineering students , 2015 .