A Convolutional Deep Self-Organizing Map Feature extraction for machine learning

In this work we propose a new Unsupervised Deep Self-Organizing Map (UDSOM) algorithm for feature extraction, quite similar to the existing multi-layer SOM architectures. The principal underlying idea of using SOMs is that if a neuron is wins n times, these n inputs that activated this neuron are similar. The basic principle consists of an alternation of phases of splitting and abstraction of regions, based on a non-linear projection of high-dimensional data over a small space using Kohonen maps following a deep architecture. The proposed architecture consists of a splitting process, layers of alternating self-organizing, a rectification function RELU and an abstraction layer (convolution-pooling). The self-organizing layer is composed of a few SOMs with each map focusing on modelling a local sub-region. The most winning neurons of each SOM are then organized in a second sampling layer to generate a new 2D map. In parallel to this transmission of the winning neurons, an abstraction of the data space is obtained after the convolution-pooling module. The ReLU is then applied. This treatment is applied more than once, changing the size of the splitting window and the displacement step on the reconstructed input image each time. In this way, local information is gathered to form more global information in the upper layers by applying each time a convolution filter of the level. The architecture of the Unsupervised Deep Self-Organizing Map is unique and retains the same principle of deep learning algorithms. This architecture can be very interesting in a Big Data environment for machine learning tasks. Experiments have been conducted to discuss how the proposed architecture shows this performance.

[1]  Alfredo Vellido.  Advances in Self-Organizing Maps, Learning Vector Quantization, Clustering and Data Visualization - Proceedings of the 13th International Workshop, WSOM+ 2019, Barcelona, Spain, June 26-28, 2019 , 2020, WSOM+.

[2]  Yihong Gong,et al.  Deep Self-Organizing Map for visual classification , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[3]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[4]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[5]  M.I. Heywood,et al.  Host-based intrusion detection using self-organizing maps , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[6]  S LewMichael,et al.  Deep learning for visual understanding , 2016 .

[7]  Adrião Duarte Dória Neto,et al.  Hierarchical and dynamic SOM applied to image compression , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[8]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[9]  Chathurika S. Wickramasinghe,et al.  Deep Self-Organizing Maps for Unsupervised Image Classification , 2019, IEEE Transactions on Industrial Informatics.

[10]  Osamu Hasegawa,et al.  Self-Organizing Incremental Neural Network (SOINN) as a Mechanism for Motor Babbling and Sensory-Motor Learning in Developmental Robotics , 2013, IWANN.

[11]  Shen Furao,et al.  A fast nearest neighbor classifier based on self-organizing incremental neural network , 2008, Neural Networks.

[12]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[13]  Thomas Brox,et al.  Unsupervised feature learning by augmenting single images , 2013, ICLR.

[14]  Michael Biehl,et al.  Prototype-based classifiers in the presence of concept drift: A modelling framework , 2019, WSOM+.

[15]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..

[16]  Mourad Zaied,et al.  Deep SOMs for automated feature extraction and classification from big data streaming , 2017, International Conference on Machine Vision.

[17]  Eugenio Culurciello,et al.  Convolutional Clustering for Unsupervised Learning , 2015, ArXiv.

[18]  Jianhua Li,et al.  A Novel Image Classification Method with CNN-XGBoost Model , 2017, IWDW.

[19]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[20]  Oliver Kramer,et al.  Self-Organizing Maps with Convolutional Layers , 2019, WSOM+.

[21]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[22]  Marc'Aurelio Ranzato,et al.  Semi-supervised learning of compact document representations with deep networks , 2008, ICML '08.

[23]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[24]  Hiroshi Dozono,et al.  Convolutional Self Organizing Map , 2016, 2016 International Conference on Computational Science and Computational Intelligence (CSCI).

[25]  Shen Furao,et al.  An incremental network for on-line unsupervised classification and topology learning , 2006, Neural Networks.

[26]  Pedro M. Domingos,et al.  Discriminative Learning of Sum-Product Networks , 2012, NIPS.

[27]  Hau-San Wong,et al.  3D head model retrieval in kernel feature space using HSOM , 2008, Pattern Recognit..

[28]  Bogdan Miclut.  Committees of Deep Feedforward Networks Trained with Few Data , 2014, GCPR.

[29]  Shen Furao,et al.  Self-Organizing Incremental Neural Network and Its Application , 2010, ICANN.

[30]  Avi Ostfeld,et al.  Protecting Water Infrastructure From Cyber and Physical Threats: Using Multimodal Data Fusion and Adaptive Deep Learning to Monitor Critical Systems , 2019, IEEE Signal Processing Magazine.

[31]  Erkki Oja,et al.  Engineering applications of the self-organizing map , 1996, Proc. IEEE.

[32]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[33]  Jiwen Lu,et al.  PCANet: A Simple Deep Learning Baseline for Image Classification? , 2014, IEEE Transactions on Image Processing.

[34]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[35]  Hideki Nakayama.  Efficient Discriminative Convolution Using Fisher Weight Map , 2013, BMVC.

[36]  Mourad Zaied,et al.  Unsupervised Features Extraction Using a Multi-view Self Organizing Map for Image Classification , 2017, 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA).

[37]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[38]  Dieter Fox,et al.  Unsupervised Feature Learning for RGB-D Based Object Recognition , 2012, ISER.

[39]  Navdeep Jaitly,et al.  Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition , 2012, INTERSPEECH.

[40]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.