Manipulator grabbing position detection with information fusion of color image and depth image using deep learning

In order to ensure stable gripping performance of manipulator in a dynamic environment, a target object grab setting model based on the candidate region suggestion network is established with the multi-target object and the anchor frame generation measurement strategy overcoming external environmental interference factors such as mutual interference between objects and changes in illumination. In which, the success rate of model detection is improved by adding small-scale anchor values for small area grabbing target position detection. Further, 94.3% crawl detection success rate is achieved on the multi-target detection data sets using the information fusion of color image and depth image. The methods in this paper effectively improve the model’s robustness and crawl success rate.

[1]  Li-Chen Fu,et al.  Grasping unknown objects using depth gradient feature with eye-in-hand RGB-D sensor , 2014, 2014 IEEE International Conference on Automation Science and Engineering (CASE).

[2]  Ying Sun,et al.  Gesture recognition algorithm based on multi-scale feature fusion in RGB-D images , 2020, IET Image Process..

[3]  Fei Zeng,et al.  Visualization of activated muscle area based on sEMG , 2020, J. Intell. Fuzzy Syst..

[4]  Ying Sun,et al.  Intelligent human computer interaction based on non redundant EMG signal , 2020, Alexandria Engineering Journal.

[5]  Siddhartha S. Srinivasa,et al.  A data-driven statistical framework for post-grasp manipulation , 2014, Int. J. Robotics Res..

[6]  Penghong Wang,et al.  A Gaussian error correction multi‐objective positioning model with NSGA‐II , 2019, Concurr. Comput. Pract. Exp..

[7]  Bo Tao,et al.  Intelligent Human-Computer Interaction Based on Surface EMG Gesture Recognition , 2019, IEEE Access.

[8]  Honghai Liu,et al.  Gesture recognition based on an improved local sparse representation classification algorithm , 2017, Cluster Computing.

[9]  Hejun Wu,et al.  Cross-Modal Attentional Context Learning for RGB-D Object Detection , 2018, IEEE Transactions on Image Processing.

[10]  Gongfa Li,et al.  CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human-Robot Interaction , 2019, Int. J. Humanoid Robotics.

[11]  Patricio A. Vela,et al.  Real-World Multiobject, Multigrasp Detection , 2018, IEEE Robotics and Automation Letters.

[12]  Di Guo,et al.  A hybrid deep architecture for robotic grasp detection , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[13]  Dong Xu,et al.  Advanced Deep-Learning Techniques for Salient and Category-Specific Object Detection: A Survey , 2018, IEEE Signal Processing Magazine.

[14]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[15]  Patricio A. Vela,et al.  Real-world Multi-object, Multi-grasp Detection , 2018 .

[16]  Saeed Khodaygan,et al.  Optimal path-planning for mobile robots to find a hidden target in an unknown environment based on machine learning , 2019, J. Ambient Intell. Humaniz. Comput..

[17]  Jinjun Chen,et al.  Differential Privacy Techniques for Cyber Physical Systems: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[18]  Bo Tao,et al.  Gesture recognition based on skeletonization algorithm and CNN with ASL database , 2018, Multimedia Tools and Applications.

[19]  Gongfa Li,et al.  A novel feature extraction method for machine learning based on surface electromyography from healthy brain , 2019, Neural Computing and Applications.

[20]  Zhihua Cui,et al.  Personalized Recommendation System Based on Collaborative Filtering for IoT Scenarios , 2020, IEEE Transactions on Services Computing.

[21]  Du Jiang,et al.  Gesture recognition based on multi‐modal feature weight , 2020, Concurr. Comput. Pract. Exp..

[22]  Dinesh Manocha,et al.  Transferring Grasp Configurations using Active Learning and Local Replanning , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[23]  Ying Sun,et al.  Surface EMG hand gesture recognition system based on PCA and GRNN , 2019, Neural Computing and Applications.

[24]  Peter Nielsen,et al.  Co-evolutionary particle swarm optimization algorithm for two-sided robotic assembly line balancing problem , 2016 .

[25]  Gongfa Li,et al.  Hand medical monitoring system based on machine learning and optimal EMG feature set , 2019, Personal and Ubiquitous Computing.

[26]  Dan Liu,et al.  Deep learning based smart radar vision system for object recognition , 2019, J. Ambient Intell. Humaniz. Comput..

[27]  Fei Zeng,et al.  Application of PSO-RBF neural network in gesture recognition of continuous surface EMG signals , 2020, J. Intell. Fuzzy Syst..

[28]  Gongfa Li,et al.  Gesture recognition based on surface electromyography‐feature image , 2020, Concurr. Comput. Pract. Exp..

[29]  Hui Yu,et al.  Gesture recognition based on binocular vision , 2018, Cluster Computing.

[30]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[31]  Jitendra Malik,et al.  Learning to Poke by Poking: Experiential Learning of Intuitive Physics , 2016, NIPS.

[32]  Hong Liu,et al.  Robot grasp detection using multimodal deep convolutional neural networks , 2016 .

[33]  Gongfa Li,et al.  Grip strength forecast and rehabilitative guidance based on adaptive neural fuzzy inference system using sEMG , 2019, Personal and Ubiquitous Computing.

[34]  Sven Behnke,et al.  RGB-D object detection and semantic segmentation for autonomous manipulation in clutter , 2018, Int. J. Robotics Res..

[35]  Ping Li,et al.  SCN: Switchable Context Network for Semantic Segmentation of RGB-D Images , 2020, IEEE Transactions on Cybernetics.

[36]  Gongfa Li,et al.  Decomposition algorithm for depth image of human health posture based on brain health , 2019, Neural Computing and Applications.

[37]  Deepak Kumar Jain,et al.  An evaluation of deep learning based object detection strategies for threat object detection in baggage security imagery , 2019, Pattern Recognit. Lett..

[38]  Ying Sun,et al.  Multi-object intergroup gesture recognition combined with fusion feature and KNN algorithm , 2020, J. Intell. Fuzzy Syst..

[39]  Hao Wu,et al.  Occlusion gesture recognition based on improved SSD , 2020, Concurr. Comput. Pract. Exp..

[40]  Li Chen,et al.  AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection , 2018, Neurocomputing.

[41]  Gongfa Li,et al.  Grasping force prediction based on sEMG signals , 2020 .

[42]  Peng Wang,et al.  A Method Combining CNN and ELM for Feature Extraction and Classification of SAR Image , 2019, J. Sensors.

[43]  Danica Kragic,et al.  Trends and challenges in robot manipulation , 2019, Science.

[44]  Christopher Kanan,et al.  Robotic grasp detection using deep convolutional neural networks , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[45]  Zhihua Cui,et al.  A Hybrid BlockChain-Based Identity Authentication Scheme for Multi-WSN , 2020, IEEE Transactions on Services Computing.

[46]  Yaoqing Weng,et al.  Enhancement of real‐time grasp detection by cascaded deep convolutional neural networks , 2020, Concurr. Comput. Pract. Exp..

[47]  Gongfa Li,et al.  Modeling of the Public Opinion Polarization Process with the Considerations of Individual Heterogeneity and Dynamic Conformity , 2019, Mathematics.

[48]  Zhihua Cui,et al.  An under‐sampled software defect prediction method based on hybrid multi‐objective cuckoo search , 2019, Concurr. Comput. Pract. Exp..

[49]  Gongfa Li,et al.  Jointly network image processing: multi-task image semantic segmentation of indoor scene based on CNN , 2020, IET Image Process..

[50]  Honghai Liu,et al.  Surface EMG data aggregation processing for intelligent prosthetic action recognition , 2018, Neural Computing and Applications.

[51]  Bowen Luo,et al.  Improvement of Maximum Variance Weight Partitioning Particle Filter in Urban Computing and Intelligence , 2019, IEEE Access.

[52]  Yumin Liao,et al.  A Quantized CNN-Based Microfluidic Lensless-Sensing Mobile Blood-Acquisition and Analysis System , 2019, Sensors.

[53]  Martijn Wisse,et al.  Fast grasping of unknown objects using principal component analysis , 2017 .

[54]  Honghai Liu,et al.  Jointly network: a network based on CNN and RBM for gesture recognition , 2018, Neural Computing and Applications.

[55]  Hao Wu,et al.  Dynamic Gesture Recognition in the Internet of Things , 2019, IEEE Access.

[56]  Xuan Song,et al.  Visual graph mining for graph matching , 2019, Comput. Vis. Image Underst..

[57]  Matei T. Ciocarlie,et al.  Contact-reactive grasping of objects with partial shape information , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[58]  Douglas Chai,et al.  Review of Deep Learning Methods in Robotic Grasp Detection , 2018, Multimodal Technol. Interact..

[59]  Honghai Liu,et al.  Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm , 2017, Cluster Computing.

[60]  Xinyu Liu,et al.  Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics , 2017, Robotics: Science and Systems.

[61]  Zhihua Cui,et al.  Hybrid many-objective particle swarm optimization algorithm for green coal production problem , 2020, Inf. Sci..

[62]  Honghai Liu,et al.  Research on gesture recognition of smart data fusion features in the IoT , 2019, Neural Computing and Applications.

[63]  Gang Wang,et al.  Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition , 2015, IEEE Transactions on Multimedia.

[64]  Joseph Redmon,et al.  Real-time grasp detection using convolutional neural networks , 2014, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[65]  Luigi Palopoli,et al.  Vision-Based Robust Path Reconstruction for Robot Control , 2014, IEEE Transactions on Instrumentation and Measurement.

[66]  Qiang Ji,et al.  The Deep Regression Bayesian Network and Its Applications: Probabilistic Deep Learning for Computer Vision , 2018, IEEE Signal Processing Magazine.

[67]  Honghai Liu,et al.  Hand gesture recognition based on convolution neural network , 2017, Cluster Computing.

[68]  Jinjun Chen,et al.  A hybrid recommendation system with many-objective evolutionary algorithm , 2020, Expert Syst. Appl..

[69]  Ying Sun,et al.  Gesture recognition based on multilevel multimodal feature fusion , 2020, Journal of Intelligent & Fuzzy Systems.

[70]  Gongfa Li,et al.  Human Lesion Detection Method Based on Image Information and Brain Signal , 2019, IEEE Access.