Multisource deep learning for situation awareness

The resurgence of interest in artificial intelligence (AI) stems from impressive deep learning (DL) performance such as hierarchical supervised training using a Convolutional Neural Network (CNN). Current DL methods should provide contextual reasoning, explainable results, and repeatable understanding that require evaluation methods. This paper discusses DL techniques using multimodal (or multisource) information that extend measures of performance (MOP). Examples of joint multi-modal learning include imagery and text, video and radar, and other common sensor types. Issues with joint multimodal learning challenge many current methods and care is needed to apply machine learning methods. Results from Deep Multimodal Image Fusion (DMIF) using Electro-optical and infrared data demonstrate performance modeling based on distance to better understand DL robustness and quality to provide situation awareness.

[1]  Erik Blasch,et al.  Data association through fusion of target track and identification sets , 2000, Proceedings of the Third International Conference on Information Fusion.

[2]  K.C. Chang,et al.  Quality of information for data fusion in net centric publish and subscribe architectures , 2005, 2005 7th International Conference on Information Fusion.

[3]  Erik Blasch,et al.  Moving object detection for vehicle tracking in Wide Area Motion Imagery using 4D filtering , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[4]  Zheng Liu,et al.  Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Dale A. Lambert.  Tradeoffs in the design of higher-level fusion systems , 2007, 2007 10th International Conference on Information Fusion.

[6]  Yufeng Zheng,et al.  Qualitative and quantitative comparisons of multispectral night vision colorization techniques , 2012, Optical Engineering.

[7]  Subrata Das.  High-Level Data Fusion , 2008 .

[8]  Rick S. Blum,et al.  Fusing synergistic information from multi-sensor images: An overview from implementation to performance assessment , 2018, Inf. Fusion.

[9]  Zheng Liu,et al.  IR2VI: Enhanced Night Environmental Perception by Unsupervised Thermal Image Translation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[10]  Erik Blasch,et al.  Measures of effectiveness for high-level fusion , 2010, 2010 13th International Conference on Information Fusion.

[11]  Genshe Chen,et al.  Cooperative space object tracking using space-based optical sensors via consensus-based filters , 2016, IEEE Transactions on Aerospace and Electronic Systems.

[12]  Lauro Snidaro,et al.  Context-Enhanced Information Fusion , 2016, Advances in Computer Vision and Pattern Recognition.

[13]  T. M. Schuck,et al.  Identification friend-or-foe (IFF) sensor uncertainties, ambiguities, deception and their application to the multi-source fusion process , 2000, Proceedings of the IEEE 2000 National Aerospace and Electronics Conference. NAECON 2000. Engineering Tomorrow (Cat. No.00CH37093).

[14]  C. Yang,et al.  Pose angular-aiding for maneuvering target tracking , 2005, 2005 7th International Conference on Information Fusion.

[15]  Zheng Liu,et al.  Statistical comparison of image fusion algorithms: Recommendations , 2017, Inf. Fusion.

[16]  Erik Blasch,et al.  Random-point-based filters: analysis and comparison in target tracking , 2015, IEEE Transactions on Aerospace and Electronic Systems.

[17]  Erik Blasch,et al.  Sensor, User, Mission (SUM) Resource Management and Their Interaction with Level 2/3 Fusion , 2006, 2006 9th International Conference on Information Fusion.

[18]  Yu Liu,et al.  A general framework for image fusion based on multi-scale transform and sparse representation , 2015, Inf. Fusion.

[19]  Erik Blasch,et al.  Combining Convolutional and Recurrent Neural Networks for Human Skin Detection , 2017, IEEE Signal Processing Letters.

[20]  Gang Wang,et al.  Deep Learning-Based Classification of Hyperspectral Data , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[21]  Genshe Chen,et al.  Space object classification using deep neural networks , 2018, 2018 IEEE Aerospace Conference.

[22]  Derek C. Rose,et al.  Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[23]  Genshe Chen,et al.  Prediction of compression-induced image interpretability degradation , 2018 .

[24]  Erik Blasch,et al.  Automatic Association of Chats and Video Tracks for Activity Learning and Recognition in Aerial Video Surveillance , 2014, Sensors.

[25]  E. Blasch,et al.  Assembling a distributed fused information-based human-computer cognitive decision making tool , 2000, IEEE Aerospace and Electronic Systems Magazine.

[26]  Erik Blasch,et al.  Encoding color information for visual tracking: Algorithms and benchmark , 2015, IEEE Transactions on Image Processing.

[27]  Xue-wen Chen,et al.  Big Data Deep Learning: Challenges and Perspectives , 2014, IEEE Access.

[28]  I. Kadar,et al.  Resource management coordination with level 2/3 fusion issues and challenges [Panel Report] , 2008, IEEE Aerospace and Electronic Systems Magazine.

[29]  Erik Blasch,et al.  Covert photo classification by deep convolutional neural networks , 2017, Machine Vision and Applications.

[30]  Haibin Ling,et al.  A Container-Based Elastic Cloud Architecture for Pseudo Real-Time Exploitation of Wide Area Motion Imagery (WAMI) Stream , 2016, Journal of Signal Processing Systems.

[31]  Erik Blasch,et al.  Performance Measures of Covariance and Information Matrices in Resource Management for Target State Estimation , 2012, IEEE Transactions on Aerospace and Electronic Systems.

[32]  Panos M. Pardalos,et al.  Optimization and Cooperative Control Strategies , 2009 .