Learning to design from humans: Imitating human designers through deep learning

Humans as designers have quite versatile problem-solving strategies. Computer agents on the other hand can access large-scale computational resources to solve certain design problems. Hence, if agents can learn from human behavior, a synergetic human-agent problem-solving team can be created. This paper presents an approach to extract human design strategies and implicit rules, purely from historical human data, and use that for design generation. A two-step framework that learns to imitate human design strategies from observation is proposed and implemented. This framework makes use of deep learning constructs to learn to generate designs without any explicit information about objective and performance metrics. The framework is designed to interact with the problem through a visual interface as humans did when solving the problem. It is trained to imitate a set of human designers by observing their design state sequences without inducing problem-specific modeling bias or extra information about the problem. Furthermore, an end-to-end agent is developed that uses this deep learning framework as its core in conjunction with image processing to map pixel-to-design moves as a mechanism to generate designs. Finally, the designs generated by a computational team of these agents are then compared with actual human data for teams solving a truss design problem. Results demonstrate that these agents are able to create feasible and efficient truss designs without guidance, showing that this methodology allows agents to learn effective design strategies.

[1]  Airong Chen,et al.  A deep Convolutional Neural Network for topology optimization with strong generalization ability , 2019, ArXiv.

[2]  Christopher McComb,et al.  Toward the Rapid Design of Engineered Systems Through Deep Neural Networks , 2018, Design Computing and Cognition '18.

[3]  Sagar Patel,et al.  3D Topology Optimization using Convolutional Neural Networks , 2018, ArXiv.

[4]  J. Cagan,et al.  Design Strategy Transfer in Cognitively-Inspired Agents , 2018, Volume 2A: 44th Design Automation Conference.

[5]  Jürgen Schmidhuber,et al.  World Models , 2018, ArXiv.

[6]  Christopher McComb,et al.  Data on the design of truss structures by teams of engineering students , 2018, Data in brief.

[7]  Julian Togelius,et al.  Artificial Intelligence and Games , 2018, Springer International Publishing.

[8]  Noam Brown,et al.  Superhuman AI for heads-up no-limit poker: Libratus beats top professionals , 2018, Science.

[9]  Andrea Serani,et al.  Deep Autoencoder for Off-Line Design-Space Dimensionality Reduction in Shape Optimization , 2018 .

[10]  Sergey Levine,et al.  Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[11]  James T. Allison,et al.  An indirect design representation for topology optimization using variational autoencoder and style transfer , 2018 .

[12]  Christopher McComb,et al.  Mining Process Heuristics From Designer Action Data via Hidden Markov Models , 2017 .

[13]  Andrea Serani,et al.  Nonlinear Methods for Design-Space Dimensionality Reduction in Shape Optimization , 2017, MOD.

[14]  Sergey Levine,et al.  One-Shot Visual Imitation Learning via Meta-Learning , 2017, CoRL.

[15]  Christopher McComb,et al.  Capturing Human Sequence-Learning Abilities in Configuration Design Tasks through Markov Chains , 2017 .

[16]  M. Fuge,et al.  Design Manifolds Capture the Intrinsic Complexity and Dimension of Design Spaces , 2017 .

[17]  Tom Schaul,et al.  Learning from Demonstrations for Real World Reinforcement Learning , 2017, ArXiv.

[18]  Christopher McComb,et al.  Utilizing Markov Chains to Understand Operation Sequencing in Design Tasks , 2017 .

[19]  Thurston Sexton,et al.  Learning an Optimization Algorithm through Human Design Iterations , 2016, 1608.06984.

[20]  Honglak Lee,et al.  Estimating and Exploring the Product Form Design Space Using Deep Generative Models , 2016, DAC 2016.

[21]  Joshua B. Tenenbaum,et al.  Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation , 2016, NIPS.

[22]  Christopher McComb,et al.  Drawing Inspiration From Human Design Teams for Better Search and Optimization: The Heterogeneous Simulated Annealing Teams Algorithm , 2016 .

[23]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[24]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[25]  Tino Stanković,et al.  Experimental design research : Approaches, perspectives, applications , 2016 .

[26]  Jonathan Cagan,et al.  Human and Computational Approaches for Design Problem-Solving , 2016 .

[27]  Levent Burak Kara,et al.  Procedural Modeling Using Autoencoder Networks , 2015, UIST.

[28]  Samuel J. Gershman,et al.  Computational rationality: A converging paradigm for intelligence in brains, minds, and machines , 2015, Science.

[29]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[32]  M. Diez,et al.  Design-space dimensionality reduction in shape optimization by Karhunen–Loève expansion , 2015 .

[33]  Joshua B. Tenenbaum,et al.  Humans predict liquid dynamics using probabilistic simulation , 2015, CogSci.

[34]  Mark Fuge,et al.  Machine Learning Algorithms for Recommending Design Methods , 2014 .

[35]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[36]  Jami J. Shah,et al.  Empirical Studies of Design Thinking: Past, Present, Future , 2013 .

[37]  Tua Björklund,et al.  Initial mental representations of design problems: Differences between experts and novices , 2013 .

[38]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Stephane Ross,et al.  Interactive Learning for Sequential Decisions and Predictions , 2013 .

[40]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[41]  Shanna R. Daly,et al.  Design Heuristics in Engineering Concept Generation , 2012 .

[42]  Pascal Vincent,et al.  Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives , 2012, ArXiv.

[43]  Hao Chen,et al.  Imaginal Thinking-Based Human-Machine Design Methodology for the Configuration of Reconfigurable Machine Tools , 2012, IEEE Transactions on Industrial Informatics.

[44]  Pieter Abbeel,et al.  Autonomous Helicopter Aerobatics through Apprenticeship Learning , 2010, Int. J. Robotics Res..

[45]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[46]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[47]  Yuehong Yin,et al.  A pipe route design methodology by imitating human imaginal thinking , 2010 .

[48]  D. Singer,et al.  What Is Set-Based Design? , 2009 .

[49]  R. Baillargeon,et al.  An Account of Infants' Physical Reasoning , 2008 .

[50]  A. Woodward,et al.  Learning and the Infant Mind , 2008 .

[51]  J. Pretz,et al.  Intuition versus analysis: Strategy and experience in complex everyday problem solving , 2008, Memory & cognition.

[52]  Ashok K. Goel,et al.  Design Computing And Cognition , 2008 .

[53]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[54]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[55]  J. Busemeyer,et al.  Learning Functional Relations Based on Experience With Input-Output Pairs by Humans and Artificial Neural Networks , 2005 .

[56]  Nigel Cross,et al.  Expertise in Design: an overview , 2004 .

[57]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[58]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[59]  Ken M. Wallace,et al.  The introduction of a design heuristics extraction method , 2002, Adv. Eng. Informatics.

[60]  Dolors Ayala,et al.  Erosion and Dilation on 2-D and 3-D Digital Images: A New Size-Independent Approach , 2001, VMV.

[61]  Aude Billard,et al.  Learning human arm movements by imitation: : Evaluation of a biologically inspired connectionist architecture , 2000, Robotics Auton. Syst..

[62]  Cristiano Castelfranchi,et al.  Proceedings of the 7th International Workshop on Intelligent Agents VII. Agent Theories Architectures and Languages , 2000 .

[63]  Koen Lamberts,et al.  Knowledge, Concepts, and Categories , 1997 .

[64]  Jonathan Baxter,et al.  Theoretical Models of Learning to Learn , 1998, Learning to Learn.

[65]  Sebastian Thrun,et al.  Learning to Learn: Introduction and Overview , 1998, Learning to Learn.

[66]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[67]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[68]  Jonathan Cagan,et al.  Simulated Annealing and the Generation of the Objective Function: A Model of Learning During Problem Solving , 1997, Comput. Intell..

[69]  M. McDaniel,et al.  Extrapolation: the sine qua non for abstraction in function learning. , 1997, Journal of experimental psychology. Learning, memory, and cognition.

[70]  Uday A. Athavankar,et al.  Mental Imagery as a Design Tool , 1997, Cybern. Syst..

[71]  Arthur C. Graesser,et al.  Is it an Agent, or Just a Program?: A Taxonomy for Autonomous Agents , 1996, ATAL.

[72]  Elizabeth S. Spelke,et al.  The development of object perception , 1995 .

[73]  J. Townsend,et al.  Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. , 1993, Psychological review.

[74]  Eric J. Johnson,et al.  The adaptive decision maker , 1993 .

[75]  John W. Payne,et al.  The adaptive decision maker: Name index , 1993 .

[76]  Gabriela Goldschmidt.  Serial sketching: visual problem solving in designing , 1992 .

[77]  R A Brooks,et al.  New Approaches to Robotics , 1991, Science.

[78]  D. Meyer,et al.  Function learning: induction of continuous stimulus-response relations. , 1991, Journal of experimental psychology. Learning, memory, and cognition.

[79]  John R. Dixon,et al.  A review of research in mechanical engineering design. Part II: Representations, analysis, and design for the life cycle , 1989 .

[80]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[81]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[82]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[83]  Robert E. Tarjan,et al.  Depth-First Search and Linear Graph Algorithms , 1972, SIAM J. Comput..

[84]  J. Carroll.  FUNCTIONAL LEARNING: THE LEARNING OF CONTINUOUS FUNCTIONAL MAPPINGS RELATING STIMULUS AND RESPONSE CONTINUA , 1963 .