ClaVis: An Interactive Visual Comparison System for Classifiers

We propose ClaVis, a visual analytics system for comparative analysis of classification models. ClaVis allows users to visually compare the performance and behavior of tens to hundreds of classifiers trained with different hyperparameter configurations. Our approach is plugin-based and classifier-agnostic and allows users to add their own datasets and classifier implementations. It provides multiple visualizations, including a multivariate ranking, a similarity map, a scatterplot that reveals correlations between parameters and scores, and a training history chart. We demonstrate the effectivity of our approach in multiple case studies for training classification models in the domain of natural language processing.

[1]  Carlos Busso,et al.  IEMOCAP: interactive emotional dyadic motion capture database , 2008, Lang. Resour. Evaluation.

[2]  Martin Wattenberg,et al.  Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow , 2018, IEEE Transactions on Visualization and Computer Graphics.

[3]  Nathalie Japkowicz,et al.  A Projection-Based Framework for Classifier Performance Evaluation , 2008, ECML/PKDD.

[4]  Yolanda Gil,et al.  Towards human-guided machine learning , 2019, IUI.

[5]  Nathalie Japkowicz,et al.  Visualizing Classifier Performance on Different Domains , 2008, 2008 20th IEEE International Conference on Tools with Artificial Intelligence.

[6]  Qingquan Song,et al.  Auto-Keras: An Efficient Neural Architecture Search System , 2018, KDD.

[7]  Hanspeter Pfister,et al.  LineUp: Visual Analysis of Multi-Attribute Rankings , 2013, IEEE Transactions on Visualization and Computer Graphics.

[8]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[9]  Desney S. Tan,et al.  EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers , 2009, CHI.

[10]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection , 2018, J. Open Source Softw..

[11]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[12]  Harald Piringer,et al.  A Partition-Based Framework for Building and Validating Regression Models , 2013, IEEE Transactions on Visualization and Computer Graphics.

[13]  Leyton-BrownKevin,et al.  Auto-WEKA 2.0 , 2017 .

[14]  Ragini Verma,et al.  CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset , 2014, IEEE Transactions on Affective Computing.

[15]  Hao Chen,et al.  Robust Learning at Noisy Labeled Medical Images: Applied to Skin Lesion Classification , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[16]  Michael C. Hout,et al.  Multidimensional Scaling , 2003, Encyclopedic Dictionary of Archaeology.

[17]  Gunther Heidemann,et al.  Inter-active learning of ad-hoc classifiers for video visual analytics , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[18]  Stefan Bruckner,et al.  Visual Parameter Space Analysis: A Conceptual Framework , 2014, IEEE Transactions on Visualization and Computer Graphics.

[19]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[20]  Silvia Miksch,et al.  Visual Methods for Analyzing Probabilistic Classification Data , 2014, IEEE Transactions on Visualization and Computer Graphics.

[21]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[22]  Bongshin Lee,et al.  Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers , 2017, IEEE Transactions on Visualization and Computer Graphics.

[23]  Kalyan Veeramachaneni,et al.  ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning , 2019, CHI.

[24]  Alex Endert,et al.  A User‐based Visual Analytics Workflow for Exploratory Model Analysis , 2019, Comput. Graph. Forum.

[25]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[26]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[27]  MunznerTamara.  A Nested Model for Visualization Design and Validation , 2009 .

[28]  Elizabeth Shriberg,et al.  The ICSI Meeting Recorder Dialog Act (MRDA) Corpus , 2004, SIGDIAL Workshop.

[29]  Cristiano Premebida,et al.  Multimodal CNN Pedestrian Classification: A Study on Combining LIDAR and Camera Data , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[30]  John J. Godfrey,et al.  SWITCHBOARD: telephone speech corpus for research and development , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[31]  Min Chen,et al.  VIS4ML: An Ontology for Visual Analytics Assisted Machine Learning , 2019, IEEE Transactions on Visualization and Computer Graphics.

[32]  Stefan Holzer,et al.  VisCoDeR: A tool for visually comparing dimensionality reduction algorithms , 2018, ESANN.

[33]  Oliver Schütze,et al.  Automatic Model Selection for Neural Networks , 2019, ArXiv.

[34]  David D. Cox,et al.  Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[35]  Isabelle Guyon,et al.  Comparison of classifier methods: a case study in handwritten digit recognition , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[36]  Chris Eliasmith,et al.  Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn , 2014, SciPy.

[37]  David Maxwell Chickering,et al.  ModelTracker: Redesigning Performance Analysis Tools for Machine Learning , 2015, CHI.

[38]  Juliana Freire,et al.  Visus: An Interactive System for Automatic Machine Learning Model Building and Curation , 2019, HILDA@SIGMOD.

[39]  Jianqiang Wang,et al.  Object Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment , 2018, IEEE Transactions on Industrial Informatics.

[40]  Tamara Munzner,et al.  The four-level nested model revisited: blocks and guidelines , 2012, BELIV '12.

[41]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[42]  Medha Katehara,et al.  Prediction Scores as a Window into Classifier Behavior , 2017, ArXiv.

[43]  Martin Wattenberg,et al.  Direct-Manipulation Visualization of Deep Networks , 2017, ArXiv.

[44]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[45]  Huamin Qu,et al.  RuleMatrix: Visualizing and Understanding Classifiers with Rules , 2018, IEEE Transactions on Visualization and Computer Graphics.

[46]  W. Torgerson.  Multidimensional scaling: I. Theory and method , 1952 .