Towards data-driven filters in Paraview

Recent progress in scientific visualization has expanded the scope of visualization from being merely a way of presentation to an analysis and discovery tool. A given visualization result is usually generated by applying a series of transformations or filters to the underlying data. Nowadays, such filters use deterministic algorithms to process the data. In this work, we aim at extending this methodology towards data-driven filters, thus filters that expose the abilities of pre-trained machine learning models to the visualization system. The use of such data-driven filters is of particular interest in fields like segmentation, classification, etc., where machine learning models regularly outperform existing algorithmic approaches. To showcase this idea, we couple Paraview, the well-known flow visualization tool, with PyTorch, a deep learning framework. Paraview is extended by plugins that allow users to load pre-trained models of their choice in the form of newly developed filters. The filters transform the input data by feeding it into the model and then provide the model’s output as input to the remaining visualization pipeline. A series of simplistic use cases for segmentation and classification on image and fluid data is presented to showcase the technical applicability of such data-driven transformations in Paraview for future complex analysis tasks.

[1]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[2]  Kwan-Liu Ma,et al.  Machine Learning to Boost the Next Generation of Visualization Technology , 2007, IEEE Computer Graphics and Applications.

[3]  Andrew J. Saykin,et al.  Interactive Machine Learning by Visualization: A Small Data Solution , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[4]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[5]  Denis Lalanne,et al.  Surveying the complementary role of automatic data analysis and visualization in knowledge discovery , 2009, VAKD '09.

[6]  Kwan-Liu Ma,et al.  Intelligent Feature Extraction and Tracking for Visualizing Large-Scale 4D Flow Simulations , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[7]  Carl-Fredrik Westin,et al.  Level Set Based Integration of Segmentation and Computational Fluid Dynamics for Flow Correction in Phase Contrast Angiography , 2002, MICCAI.

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Yali Jia,et al.  Automated Segmentation of Retinal Fluid Volumes From Structural and Angiographic Optical Coherence Tomography Using Deep Learning , 2020, Translational vision science & technology.

[11]  Kwan-Liu Ma,et al.  An intelligent system approach to higher-dimensional classification of volume data , 2005, IEEE Transactions on Visualization and Computer Graphics.

[12]  Gopi Battineni,et al.  Machine learning in medicine: Performance calculation of dementia prediction by support vector machines (SVM) , 2019, Informatics in Medicine Unlocked.

[13]  Matthijs Douze,et al.  Deep Clustering for Unsupervised Learning of Visual Features , 2018, ECCV.

[14]  Nan Zheng,et al.  Digital Image Semantic Segmentation Algorithms: A Survey , 2019, J. Inf. Hiding Multim. Signal Process..

[15]  Aurélien Géron,et al.  Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems , 2017 .

[16]  Theresa-Marie Rhyne,et al.  Scientific Visualization , 1997, Encyclopedia of GIS.

[17]  Hans C. van Houwelingen,et al.  The Elements of Statistical Learning, Data Mining, Inference, and Prediction. Trevor Hastie, Robert Tibshirani and Jerome Friedman, Springer, New York, 2001. No. of pages: xvi+533. ISBN 0‐387‐95284‐5 , 2004 .

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Milan Sonka,et al.  3D Slicer as an image computing platform for the Quantitative Imaging Network. , 2012, Magnetic resonance imaging.

[20]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[21]  K. Jarrod Millman,et al.  Array programming with NumPy , 2020, Nat..

[22]  Mohammed Nouari,et al.  Honeycomb Core Milling Diagnosis using Machine Learning in the Industry 4.0 Framework , 2018, 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA).

[23]  I. Sass,et al.  CobWeb 1.0: machine learning toolbox for tomographic imaging , 2020 .

[24]  William J. Schroeder,et al.  The Visualization Toolkit , 2005, The Visualization Handbook.

[25]  Nils Thürey,et al.  Turbulent fluids , 2013, SIGGRAPH '13.

[26]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[27]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[28]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[29]  Trilce Estrada,et al.  Tensorview: visualizing the training of convolutional neural network using paraview , 2017 .

[30]  A. A. Samarskii,et al.  On numerical simulation in fluid dynamics , 1979 .

[31]  Jan Mikulka,et al.  A Novel Tool for Supervised Segmentation Using 3D Slicer , 2018, Symmetry.

[32]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[33]  I. Kohane,et al.  Big Data and Machine Learning in Health Care. , 2018, JAMA.

[34]  Heidrun Schumann,et al.  Visualization Pipeline , 2009, Encyclopedia of Database Systems.

[35]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[36]  C. C. Law,et al.  ParaView: An End-User Tool for Large-Data Visualization , 2005, The Visualization Handbook.

[37]  Samuel Leventhal,et al.  PAVE: An In Situ Framework for Scientific Visualization and Machine Learning Coupling , 2019, 2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-5).

[38]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.