Visualizing Examples of Deep Neural Networks at Scale

Many programmers want to use deep learning due to its superior accuracy in many challenging domains. Yet our formative study with ten programmers indicated that, when constructing their own deep neural networks (DNNs), they often had a difficult time choosing appropriate model structures and hyperparameter values. This paper presents ExampleNet—a novel interactive visualization system for exploring common and uncommon design choices in a large collection of open-source DNN projects. ExampleNet provides a holistic view of the distribution over model structures and hyperparameter settings in the corpus of DNNs, so users can easily filter the corpus down to projects tackling similar tasks and compare design choices made by others. We evaluated ExampleNet in a within-subjects study with sixteen participants. Compared with the control condition (i.e., online search), participants using ExampleNet were able to inspect more online examples, make more data-driven design decisions, and make fewer design mistakes.

[1]  Keith Andrews,et al.  Visual Graph Comparison , 2009, 2009 13th International Conference Information Visualisation.

[2]  Martin Wattenberg,et al.  Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow , 2018, IEEE Transactions on Visualization and Computer Graphics.

[3]  Ference Marton,et al.  Necessary Conditions of Learning , 2014 .

[4]  Maneesh Agrawala,et al.  Searching the Visual Style and Structure of D3 Visualizations , 2019, IEEE Transactions on Visualization and Computer Graphics.

[5]  Philip J. Guo,et al.  Two studies of opportunistic programming: interleaving web foraging, learning, and writing code , 2009, CHI.

[6]  Cristina V. Lopes,et al.  How Well Do Search Engines Support Code Retrieval on the Web? , 2011, TSEM.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[9]  Ronen I. Brafman,et al.  Designing with interactive example galleries , 2010, CHI.

[10]  Elena L. Glassman,et al.  Interactive Extraction of Examples from Existing Code , 2018, CHI.

[11]  John Zimmerman,et al.  Investigating How Experienced UX Designers Effectively Work with Machine Learning , 2018, Conference on Designing Interactive Systems.

[12]  Miryung Kim,et al.  Enabling Data-Driven API Design with Community Usage Data: A Need-Finding Study , 2020, CHI.

[13]  B. Berg.  Qualitative Research Methods for the Social Sciences , 1989 .

[14]  Alexander M. Rush,et al.  LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks , 2016, IEEE Transactions on Visualization and Computer Graphics.

[15]  James A. Landay,et al.  Investigating statistical machine learning as a tool for software development , 2008, CHI.

[16]  Kim Halskov,et al.  UX Design Innovation: Challenges for Working with Machine Learning as a Design Material , 2017, CHI.

[17]  Iryna Gurevych,et al.  Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks , 2017, ArXiv.

[18]  Miryung Kim,et al.  Visualizing API Usage Examples at Scale , 2018, CHI.

[19]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[20]  Paul A. Fontelo,et al.  Utilization of the PICO framework to improve searching PubMed for clinical questions , 2007, BMC Medical Informatics Decis. Mak..

[21]  Martin Wattenberg,et al.  Direct-Manipulation Visualization of Deep Networks , 2017, ArXiv.

[22]  Björn Hartmann,et al.  Delta: a tool for representing and comparing workflows , 2012, CHI.

[23]  Philip J. Guo,et al.  Software Developers Learning Machine Learning: Motivations, Hurdles, and Desires , 2019, 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[24]  Kaiyong Zhao,et al.  AutoML: A Survey of the State-of-the-Art , 2019, Knowl. Based Syst..

[25]  Katsuro Inoue,et al.  How do developers utilize source code from stack overflow? , 2018, Empirical Software Engineering.

[26]  W. Pearson.  Searching protein sequence libraries: comparison of the sensitivity and selectivity of the Smith-Waterman and FASTA algorithms. , 1991, Genomics.

[27]  Ranjitha Kumar,et al.  Webzeitgeist: design mining the web , 2013, CHI.

[28]  Kathryn T. Stolee,et al.  How developers search for code: a case study , 2015, ESEC/SIGSOFT FSE.

[29]  Michael I. Jordan,et al.  How Does Learning Rate Decay Help Modern Neural Networks , 2019 .

[30]  Qian Yang,et al.  Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models , 2018, Conference on Designing Interactive Systems.

[31]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .