Active Visual-Tactile Cross-Modal Matching

Tactile and visual modalities frequently occur in cognitive robotics. Their matching problem is of highly interesting in many practical scenarios since it provides different properties about objects. In this paper, we investigate the active visual-tactile cross-modal matching problem which is formulated as retrieving the relevant sample in unlabeled gallery visual dataset in response to the tactile query sample. Such a problem exhibits a nontrivial challenge that there does not exist sample-to-sample pairing relation between tactile and visual modalities. To this end, we design a shared dictionary learning model which can simultaneously learn the projection subspace and the latent shared dictionary for the visual and tactile measurements. In addition, an optimization algorithm is developed to effectively solve the shared dictionary learning problem. Based on the obtained solution, the visual-tactile cross-modal matching algorithm can be easily developed. Finally, we perform experimental validations on the PHAC-2 datasets to show the effectiveness of the proposed visual-tactile cross-modal matching framework and method.

[1]  Yan Peng,et al.  Adaptive Impedance Control of Human–Robot Cooperation Using Reinforcement Learning , 2017, IEEE Transactions on Industrial Electronics.

[2]  Deborah H. Lester,et al.  Internet Shopping and Buying Behavior of College Students , 2005 .

[3]  Francesco Massi,et al.  Tactile Perception and Friction-Induced Vibrations: Discrimination of Similarly Patterned Wood-Like Surfaces , 2017, IEEE Transactions on Haptics.

[4]  Di Guo,et al.  Object Recognition Using Tactile Measurements: Kernel Sparse Coding Methods , 2016, IEEE Transactions on Instrumentation and Measurement.

[5]  Eris Chinellato,et al.  Visual neuroscience of robotic grasping , 2008 .

[6]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[7]  Wenming Zheng,et al.  Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[8]  Qinghua Zheng,et al.  An Adaptive Semisupervised Feature Analysis for Video Semantic Recognition , 2018, IEEE Transactions on Cybernetics.

[9]  Christoph H. Lampert,et al.  Learning Dynamic Tactile Sensing With Robust Vision-Based Training , 2011, IEEE Transactions on Robotics.

[10]  Yueting Zhuang,et al.  Supervised Coupled Dictionary Learning with Group Structures for Multi-modal Retrieval , 2013, AAAI.

[11]  Wei Wang,et al.  A Comprehensive Survey on Cross-modal Retrieval , 2016, ArXiv.

[12]  Gordon Cheng,et al.  A Tactile-Based Framework for Active Object Learning and Discrimination using Multimodal Robotic Skin , 2017, IEEE Robotics and Automation Letters.

[13]  Yang Gao,et al.  Deep learning for tactile understanding from visual and haptic data , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Qinghua Zheng,et al.  Adaptive Unsupervised Feature Selection With Structure Regularization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Christoph H. Lampert,et al.  Weakly-Paired Maximum Covariance Analysis for Multimodal Dimensionality Reduction and Transfer Learning , 2010, ECCV.

[16]  Terry L. Childers,et al.  Individual Differences in Haptic Information Processing: The "Need for Touch" Scale , 2003 .

[17]  Eckehard G. Steinbach,et al.  Multimodal Feature-Based Surface Material Classification , 2017, IEEE Transactions on Haptics.

[18]  Nadja Althaus,et al.  Modeling Cross-Modal Interactions in Early Word Learning , 2013, IEEE Transactions on Autonomous Mental Development.

[19]  Qinghua Hu,et al.  Kernel sparse representation for time series classification , 2015, Inf. Sci..

[20]  Di Guo,et al.  Extreme Kernel Sparse Learning for Tactile Object Recognition , 2017, IEEE Transactions on Cybernetics.

[21]  Jean-Claude Martin,et al.  Combining Facial Expression and Touch for Perceiving Emotional Valence , 2018, IEEE Transactions on Affective Computing.

[22]  Nikhil Rasiwasia,et al.  Cluster Canonical Correlation Analysis , 2014, AISTATS.

[23]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[24]  Fuchun Sun,et al.  Kernel Regularized Nonlinear Dictionary Learning for Sparse Coding , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[25]  Fuchun Sun,et al.  Visual–Tactile Fusion for Object Recognition , 2017, IEEE Transactions on Automation Science and Engineering.

[26]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[27]  Gert Kootstra,et al.  Design of a flexible tactile sensor for classification of rigid and deformable objects , 2014, Robotics Auton. Syst..

[28]  Lei Zhang,et al.  Metaface learning for sparse representation based face recognition , 2010, 2010 IEEE International Conference on Image Processing.

[29]  Di Guo,et al.  Recent progress on tactile object recognition , 2017 .

[30]  Yu-Chiang Frank Wang,et al.  Coupled Dictionary and Feature Space Learning with Applications to Cross-Domain Image Synthesis and Recognition , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  Angel P. del Pobil,et al.  The Visual Neuroscience of Robotic Grasping: Achieving Sensorimotor Skills through Dorsal-Ventral Stream Integration , 2015 .

[32]  R. Klatzky,et al.  Haptic exploration in the presence of vision. , 1993, Journal of experimental psychology. Human perception and performance.

[33]  Michael Elad,et al.  Linearized Kernel Dictionary Learning , 2015, IEEE Journal of Selected Topics in Signal Processing.

[34]  Rama Chellappa,et al.  Coupled Projections for Adaptation of Dictionaries , 2015, IEEE Transactions on Image Processing.

[35]  Yang Yang,et al.  Adversarial Cross-Modal Retrieval , 2017, ACM Multimedia.

[36]  Marco Cuturi,et al.  Fast Global Alignment Kernels , 2011, ICML.

[37]  Charles C. Kemp,et al.  A CRF that combines touch and vision for haptic mapping , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[38]  S. Forsythe,et al.  Adoption of sensory enabling technology for online apparel shopping , 2009 .

[39]  Xingang Zhao,et al.  Human Cooperative Wheelchair With Brain–Machine Interaction Based on Shared Control Strategy , 2017, IEEE/ASME Transactions on Mechatronics.

[40]  Yu Kang,et al.  Human–Robot Coordination Control of Robotic Exoskeletons by Skill Transfers , 2017, IEEE Transactions on Industrial Electronics.

[41]  Gunnar Blohm,et al.  EEG-Based Perceived Tactile Location Prediction , 2015, IEEE Transactions on Autonomous Mental Development.

[42]  Amir Amedi,et al.  Multisensory visual–tactile object related network in humans: insights gained using a novel crossmodal adaptation approach , 2009, Experimental Brain Research.

[43]  Trevor Darrell,et al.  Robotic learning of haptic adjectives through physical interaction , 2015, Robotics Auton. Syst..

[44]  Barbara Caputo,et al.  Using Object Affordances to Improve Object Recognition , 2011, IEEE Transactions on Autonomous Mental Development.

[45]  Devraj Mandal,et al.  Generalized Coupled Dictionary Learning Approach With Applications to Cross-Modal Matching , 2016, IEEE Transactions on Image Processing.

[46]  Véronique Perdereau,et al.  Tactile sensing in dexterous robot hands - Review , 2015, Robotics Auton. Syst..

[47]  Fuchun Sun,et al.  Seeing by touching: Cross-modal matching for tactile and vision measurements , 2017, 2017 2nd International Conference on Advanced Robotics and Mechatronics (ICARM).