Deep Learning: A Critical Appraisal

Although deep learning has historical roots going back decades, neither the term "deep learning" nor the approach was popular just over five years ago, when the field was reignited by papers such as Krizhevsky, Sutskever and Hinton's now classic (2012) deep network model of Imagenet. What has the field discovered in the five subsequent years? Against a background of considerable progress in areas such as speech recognition, image recognition, and game playing, and considerable enthusiasm in the popular press, I present ten concerns for deep learning, and suggest that deep learning must be supplemented by other techniques if we are to reach artificial general intelligence.

[1]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[2]  Chris Dyer,et al.  The NarrativeQA Reading Comprehension Challenge , 2017, TACL.

[3]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[4]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[6]  Sameer Singh,et al.  “Why Should I Trust You?”: Explaining the Predictions of Any Classifier , 2016, NAACL.

[7]  Peter M. Vishton,et al.  Rule learning by seven-month-old infants. , 1999, Science.

[8]  Ernest Davis,et al.  Commonsense reasoning and commonsense knowledge in artificial intelligence , 2015, Commun. ACM.

[9]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[10]  Sarah-Jane Leslie,et al.  Children's interpretations of general quantifiers, specific quantifiers and generics , 2015, Language, cognition and neuroscience.

[11]  Bhaskara Marthi,et al.  A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs , 2017, Science.

[12]  Charles L. Ortiz.  Why We Need a Physically Embodied Turing Test and What It Might Look Like , 2016, AI Mag..

[13]  Dawn Song,et al.  Robust Physical-World Attacks on Deep Learning Models , 2017, 1707.08945.

[14]  François Chollet,et al.  Deep Learning with Python , 2017 .

[15]  Martín Abadi,et al.  Learning a Natural Language Interface with Neural Programmer , 2016, ICLR.

[16]  Thomas L. Dean,et al.  The atoms of neural computation , 2014, Science.

[17]  Razvan Pascanu,et al.  Visual Interaction Networks: Learning a Physics Simulator from Video , 2017, NIPS.

[18]  Percy Liang,et al.  Adversarial Examples for Evaluating Reading Comprehension Systems , 2017, EMNLP.

[19]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[20]  G. Marcus.  The Algebraic Mind: Integrating Connectionism and Cognitive Science , 2001 .

[21]  H. Gardner,et al.  Frames of Mind: The Theory of Multiple Intelligences , 1983 .

[22]  Yoshua Bengio,et al.  Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Klaus-Robert Müller,et al.  Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models , 2017, ArXiv.

[24]  Mark Steedman,et al.  Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , 2012 .

[25]  Tim Rocktäschel,et al.  Programming with a Differentiable Forth Interpreter , 2016, ICML.

[26]  Michael R. Genesereth,et al.  General Game Playing: Overview of the AAAI Competition , 2005, AI Mag..

[27]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[28]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[29]  W. B. Roberts,et al.  Machine Learning: The High Interest Credit Card of Technical Debt , 2014 .

[30]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Cathy O'Neil,et al.  Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy , 2016, Vikalpa: The Journal for Decision Makers.

[32]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[33]  G. Marcus.  Kluge: The Haphazard Construction of the Human Mind , 2008 .

[34]  Bernhard Schölkopf,et al.  Discovering Causal Signals in Images , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Iris Berent,et al.  Binding at Birth: The Newborn Brain Detects Identity Relations and Sequential Position in Speech , 2012, Journal of Cognitive Neuroscience.

[36]  Apurv Jain.  Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy , 2017, Business Economics.

[37]  Dileep George,et al.  Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics , 2017, ICML.

[38]  Rob Fergus,et al.  Learning Physical Intuition of Block Towers by Example , 2016, ICML.

[39]  D. Lazer,et al.  The Parable of Google Flu: Traps in Big Data Analysis , 2014, Science.

[40]  Oren Etzioni,et al.  Moving beyond the Turing Test with the Allen AI Science Challenge , 2016, Commun. ACM.

[41]  R. Needham,et al.  Artificial Intelligence : A General Survey , 2012 .

[42]  Ernest Davis,et al.  How to Write Science Questions that Are Easy for People and Hard for Computers , 2016, AI Mag..

[43]  D. Kahneman.  Thinking, Fast and Slow , 2011 .

[44]  Yann LeCun,et al.  Predicting Deeper into the Future of Semantic Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[45]  Katherine D. Kinzler,et al.  Core knowledge. , 2007, Developmental science.

[46]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[47]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Jiajun Wu,et al.  Learning to See Physics via Visual De-animation , 2017, NIPS.

[49]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Kai-Uwe Kühnberger,et al.  Neural-Symbolic Learning and Reasoning: A Survey and Interpretation , 2017, Neuro-Symbolic Artificial Intelligence.

[51]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[52]  Itamar Arel,et al.  Beyond the Turing Test , 2009, Computer.

[53]  Marco Baroni,et al.  Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks , 2017, ICLR 2018.

[54]  Ernest Davis,et al.  Commonsense reasoning about containers using radically incomplete information , 2017, Artif. Intell..

[55]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[56]  Randy H. Katz,et al.  A Berkeley View of Systems Challenges for AI , 2017, ArXiv.

[57]  Yann LeCun,et al.  Generalization and network design strategies , 1989 .

[58]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[59]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[60]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[61]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[62]  Marc'Aurelio Ranzato,et al.  Building high-level features using large scale unsupervised learning , 2011, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[63]  Andrew Y. Ng,et al.  Semantic Compositionality through Recursive Matrix-Vector Spaces , 2012, EMNLP.

[64]  G. Marcus.  Rethinking Eliminative Connectionism , 1998, Cognitive Psychology.

[65]  Joshua B. Tenenbaum,et al.  Building machines that learn and think like people , 2016, Behavioral and Brain Sciences.

[66]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[67]  S Pinker,et al.  Overregularization in language acquisition. , 1992, Monographs of the Society for Research in Child Development.

[68]  G. Marcus.  Can connectionism save constructivism? , 1998, Cognition.

[69]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[70]  Praveen Paritosh,et al.  Toward a Comprehension Challenge, Using Crowdsourcing as a Tool , 2016, AI Mag..

[71]  Logan Engstrom,et al.  Synthesizing Robust Adversarial Examples , 2017, ICML.

[72]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[73]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.