Criminisi Algorithm Applied to a GelSight Fingertip Sensor for Multi-modality Perception

Tactile sensing is an important perception mode for a robot to interact with the external environment and perform tasks such as object manipulation. However, there are still limitations for current artificial tactile sensing technologies to perform manipulation tasks as well as a human. In this paper, we propose a biomimetic tactile sensor based on the principle of the GelSight fingertip tactile sensor [1], but following a different architecture design and using a different image processing algorithm. The contribution of our proposed sensor is to concurrently detect contact force and rebuild the object shape. In this paper, we focus on generalizing the tactile-related information from the force-related information by using the Criminisi algorithm [2]. Concerning the hardware, we use a combination of a coated silicone elastomer (with a marker array across its surface) and an internal spring structure to determine force information. A camera placed at the bottom of the sensor housing captures the deformation of the elastomer surface illuminated by an internal light. With regards to the software, we analyze the position and shape changes of the marker array to distinguish different types of contact forces, however, the use of the markers causes interference in the shape-recognition step, so we apply the Criminisi algorithm to remove the force markers from the captured images and restore the object contour image. We use a Convolutional Neural Network (CNN) to determine the shape of the handled object based on the captured deformation; we achieve an accuracy of 89.6% for shape classification.