Feature Engineering for Design Thinking Assessment

As design and design thinking become increasingly important competencies for a modern workforce, the burden of assessing these fuzzy skills creates a scalability bottleneck. Toward addressing this need, this paper presents an exploratory study into a scalable computational approach for design thinking assessment. In this study, student responses to a variety of contextualized design questions ? gathered both before and after participation in a design thinking training course ? are analyzed. Specifically, a variety of text features are engineered, tested, and interpreted within a design thinking framework in order to identify specific markers of design thinking skill acquisition. Key findings of this work include identification of text features that may enable scalable measurement of (1) user-centric language and (2) design thinking concept acquisition. These results contribute toward the creation of computational tools to ease the burden of providing feedback about design thinking skills to a wide audience.

[1]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[2]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[3]  Pedro M. Domingos A few useful things to know about machine learning , 2012, Commun. ACM.

[4]  Martin Tomitsch,et al.  Design Thinking Education: A Comparison of Massive Open Online Courses , 2018 .

[5]  Marleen Davis,et al.  First Year Design Studio , 1982 .

[6]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[7]  Cara Wrigley,et al.  Design Thinking pedagogy: the Educational Design Ladder , 2015 .

[8]  Petr Sojka,et al.  Software Framework for Topic Modelling with Large Corpora , 2010 .

[9]  Paulo Blikstein,et al.  Towards a formal assessment of design literacy: Analyzing K-12 students' stance towards inquiry , 2016 .

[10]  Richard Kimbell,et al.  Evolving project e-scape for national assessment , 2012 .

[11]  Leidy E. Klotz,et al.  Using survey questions to identify and learn more about those who exhibit design thinking traits , 2015 .

[12]  G. Harry McLaughlin,et al.  SMOG Grading - A New Readability Formula. , 1969 .

[13]  Susan McLaren,et al.  Exploring key discriminators of progression: relationships between attitude, meta-cognition and performance of novice designers at a time of transition , 2008 .

[14]  Scott R. Bartholomew,et al.  Assessing Open-Ended Design Problems. , 2017 .

[15]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[16]  Nigel Cross,et al.  Design Thinking: Understanding How Designers Think and Work , 2011 .

[17]  Stephen P. Balfour,et al.  Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review™. , 2013 .

[18]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[19]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[20]  A. Royalty,et al.  Destination, Imagination and the Fires Within: Design Thinking in a Middle School Classroom , 2010 .

[21]  Tristan Miller,et al.  Essay Assessment with Latent Semantic Analysis , 2003 .

[22]  David H. Jonassen,et al.  Everyday Problem Solving In Engineering: Lessons For Educators , 2005 .

[23]  David H. Jonassen,et al.  Everyday Problem Solving in Engineering: Lessons for Engineering Educators , 2006 .

[24]  Karen Spärck Jones A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[25]  Susan Stewart,et al.  Design Thinking: Understanding How Designers Think and Work , 2011 .

[26]  Scott R. Bartholomew Assessing Open-Ended Design Problems: An Emphasis on ACJ, as a Promising Design-Assessment Approach, Will Strengthen Our Position and Potential as a Vital Element of K-12 Education Moving Forward , 2017 .

[27]  Thomas Kelley,et al.  Creative Confidence: Unleashing the Creative Potential Within Us All , 2013 .

[28]  Denny Davis,et al.  Engineering Design Assessment Processes and Scoring Scales for Program Improvement and Accountability , 2002 .

[29]  Karen Sparck Jones A statistical interpretation of term specificity and its application in retrieval , 1972 .

[30]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[31]  Kristin L. Wood,et al.  Design Innovation: A Study of Integrated Practice , 2017 .

[32]  Yaron Doppelt,et al.  Assessing creative thinking in design-based learning , 2009 .

[33]  Carman Neustaedter,et al.  Becoming a Design Thinker: Assessing the Learning Process of Students in a Secondary Level Design Thinking Course. , 2018 .

[34]  Yasemin Afacan,et al.  Assessing creativity in design education: Analysis of creativity factors in the first-year design studio , 2012 .

[35]  Steven Beyerlein,et al.  Assessment Framework For Capstone Design Courses , 2006 .