Commuting Conditional GANS for Multi-Modal Fusion

This paper presents a data driven approach to multi-modal fusion where a hidden latent sub-space between the different modalities is learned. The hidden space is estimated via a bank of Conditional GANs which also commute with each other, leading to an output that lies in a common subspace. Experimental results show improved detection performance compared to existing fusion techniques in ideal as well as noisy sensor condition.

[1]  P.K. Varshney,et al.  On the Detection of Footsteps Based on Acoustic and Seismic Sensing , 2007, 2007 Conference Record of the Forty-First Asilomar Conference on Signals, Systems and Computers.

[2]  Zhi-Quan Luo,et al.  Event Driven Fusion , 2019, 1904.11520.

[3]  Thyagaraju Damarla,et al.  Personnel and Vehicle Data Collection at Aberdeen Proving Ground (APG) and its Distribution for Research , 2015 .

[4]  Trevor Darrell,et al.  Learning with Side Information through Modality Hallucination , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Rauf Izmailov,et al.  Learning using privileged information: similarity control and knowledge transfer , 2015, J. Mach. Learn. Res..

[6]  G. Frobenius,et al.  Ueber lineare Substitutionen und bilineare Formen. , 2022 .

[7]  Paolo Glorioso.  On common eigenbases of commuting operators , 2012 .

[8]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[9]  Liyi Dai,et al.  Cross-Modality Distillation: A Case for Conditional Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Asok Ray,et al.  Multimodal Task-Driven Dictionary Learning for Image Classification , 2015, IEEE Transactions on Image Processing.

[11]  Fakhri Karray,et al.  Multisensor data fusion: A review of the state-of-the-art , 2013, Inf. Fusion.

[12]  Zhi-Quan Luo,et al.  Decision Level Fusion: An Event Driven Approach , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[13]  Zhi-Quan Luo,et al.  Convex optimization approach to identify fusion for multisensor target tracking , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[14]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[15]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[16]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[17]  Pascal Vasseur,et al.  Introduction to Multisensor Data Fusion , 2005, The Industrial Information Technology Handbook.

[18]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[19]  Erik Skau,et al.  Fusing Heterogeneous Data: A Case for Remote Sensing and Social Media , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Jon Gauthier.  Conditional generative adversarial nets for convolutional face generation , 2015 .

[21]  Lingjie Li.  DATA FUSION AND FILTERING FOR TARGET TRACKING AND IDENTIFICATION , 2003 .