A Deep Learning Based Approach to Predict Sequential Design Decisions

During a design process, designers iteratively go back and forth between different design stages to explore the design space and search for the best design solution that satisfies all design constraints. For complex design problems, human has shown surprising capability in effectively reducing the dimensionality of design space and quickly converging it to a reasonable range for algorithms to step in and continue the search process. Therefore, modeling how human designers make decisions in such a sequential design process can help discover beneficial design patterns, strategies, and heuristics, which are important to the development of new algorithms embedded with human intelligence to augment computational design. In this paper, we develop a deep learning based approach to model and predict designers’ sequential decisions in a system design context. The core of this approach is an integration of the function-behavior-structure model for design process characterization and the long short term memory unit model for deep leaning. This approach is demonstrated in a solar energy system design case study, and its prediction accuracy is evaluated benchmarked on several commonly used models for sequential design decisions, such as Markov Chain model, Hidden Markov Chain model, and random sequence generation model. The results indicate that the proposed approach outperforms the other traditional models. This implies that during a system design task, designers are very likely to reply on both short-term and long-term memory of past design decisions in guiding their decision making in future design process. Our approach is general to be applied in many other design contexts as long as the sequential design action data is available.

[1]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[2]  John S. Gero,et al.  Design Prototypes: A Knowledge Representation Schema for Design , 1990, AI Mag..

[3]  Anthony J. Robinson,et al.  An application of recurrent nets to phone probability estimation , 1994, IEEE Trans. Neural Networks.

[4]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[5]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Robert P. Smith,et al.  A predictive model of sequential iteration in engineering design , 1997 .

[8]  William P. Anthony,et al.  The Influence of Tacit Knowledge and Collective Mind on Strategic Planning , 1998 .

[9]  Guy M. Nicoletti AN ANALYSIS OF NEURAL NETWORKS AS SIMULATORS AND EMULATORS , 2000, Cybern. Syst..

[10]  Andrew G. Barto,et al.  Optimal learning: computational procedures for bayes-adaptive markov decision processes , 2002 .

[11]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[12]  Michel Verleysen,et al.  The Curse of Dimensionality in Data Mining and Time Series Prediction , 2005, IWANN.

[13]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[14]  Jürgen Schmidhuber,et al.  Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks , 2007, NIPS.

[15]  Jeff W T Kan,et al.  Using the FBS Ontology to Capture Semantic Design Information in Design Protocol Studies , 2007 .

[16]  Tyson R. Browning,et al.  Design Process Sequencing With Competent Genetic Algorithms , 2007 .

[17]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[18]  Somnuk Phon-Amnuaisuk,et al.  A comparison of HMM, Naïve Bayesian, and Markov model in exploiting knowledge content in digital ink: A case study on handwritten music notation recognition , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[19]  Nicky J Welton,et al.  Exploring the Research Decision Space: The Expected Value of Information for Sequential Research Designs , 2010, Medical decision making : an international journal of the Society for Medical Decision Making.

[20]  Christian Wolf,et al.  Sequential Deep Learning for Human Action Recognition , 2011, HBU.

[21]  A. V. Olgac,et al.  Performance Analysis of Various Activation Functions in Generalized MLP Architectures of Neural Networks , 2011 .

[22]  Geoffrey E. Hinton,et al.  Acoustic Modeling Using Deep Belief Networks , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[23]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[24]  Mohammad Soleymani,et al.  Continuous emotion detection using EEG signals and facial expressions , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[25]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[26]  CHARLES XIE,et al.  Time series analysis method for assessing engineering design processes using a CAD tool , 2014 .

[27]  Byunghan Lee,et al.  DNA-Level Splice Junction Prediction using Deep Recurrent Neural Networks , 2015, ArXiv.

[28]  John S. Gero,et al.  An Empirical Foundation for Design Patterns in Parametric Design , 2015 .

[29]  Jitesh H. Panchal,et al.  Behavioral Experimentation and Game Theory in Engineering Systems Design , 2015 .

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[32]  Yi Li,et al.  Gene expression inference with deep learning , 2015, bioRxiv.

[33]  Yang Xiang,et al.  A Two Phase Deep Learning Model for Identifying Discrimination from Tweets , 2016, EDBT.

[34]  Thurston Sexton,et al.  Learning an Optimization Algorithm through Human Design Iterations , 2016, 1608.06984.

[35]  Kedar Potdar,et al.  A Comparative Study of Categorical Variable Encoding Techniques for Neural Network Classifiers , 2017 .

[36]  Jerzy Korczak,et al.  Deep learning for financial time series forecasting in A-Trader system , 2017, 2017 Federated Conference on Computer Science and Information Systems (FedCSIS).

[37]  Christopher McComb,et al.  Mining Process Heuristics From Designer Action Data via Hidden Markov Models , 2017 .

[38]  Daniel Sánchez Santolaya Using recurrent neural networks to predict customer behavior from interaction data , 2017 .

[39]  Christopher McComb,et al.  Capturing Human Sequence-Learning Abilities in Configuration Design Tasks through Markov Chains , 2017 .

[40]  Jitesh H. Panchal,et al.  Understanding Design Decisions Under Competition Using Games With Information Acquisition and a Behavioral Experiment , 2017 .

[41]  Boris Hanin,et al.  Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations , 2017, Mathematics.

[42]  Jie Chao,et al.  Learning and teaching engineering design through modeling and simulation on a CAD platform , 2018, Comput. Appl. Eng. Educ..

[43]  François Chollet,et al.  Deep Learning mit Python und Keras , 2018 .

[44]  Fanglan Zheng,et al.  Sequential Behavioral Data Processing Using Deep Learning and the Markov Transition Field in Online Fraud Detection , 2018, ArXiv.

[45]  Fei Wang,et al.  Deep learning for healthcare: review, opportunities and challenges , 2018, Briefings Bioinform..

[46]  Zhenghui Sha,et al.  Automatic Clustering of Sequential Design Behaviors , 2018 .