Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures

We present a framework for the acquisition and parametrization of object material properties. The introduced acquisition device, denoted as Texplorer2, is able to extract surface material properties while a human operator is performing exploratory procedures. Using the Texplorer2, we scanned 184 material classes which we labeled according to biological, chemical, and geological naming conventions. Based on these real material recordings, we introduce a novel set of mathematical features which align with corresponding material properties defined in perceptual studies from related work and classify the materials using common machine learning techniques. Validation results of the proposed multi-modal features lead to an overall classification accuracy of 90.2% <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 1.2% and an F<inline-formula><tex-math notation="LaTeX">$_\text{\text{1}}$</tex-math></inline-formula> score of 0.90 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 0.01 using the random forest classifier. For the sake of comparison, a deep neural network is trained and tested on images of the material surfaces; it outperforms (90.7% <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 1.0%) the hand-crafted feature-based approach yet leads to more critical misclassifications in terms of the proposed taxonomy.

[1]  Yang Gao,et al.  Proton: A visuo-haptic data acquisition system for robotic learning of surface properties , 2016, 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[2]  Theodoros Giannakopoulos,et al.  Introduction to audio analysis , 2016 .

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  A. Kappers,et al.  Tactile perception of thermal diffusivity , 2009, Attention, perception & psychophysics.

[5]  J. F. Dammann,et al.  Temporal Frequency Channels Are Linked across Audition and Touch , 2009, Current Biology.

[6]  Cagatay Basdogan,et al.  Haptic Rendering in Virtual Environments , 2002 .

[7]  Allison M. Okamura,et al.  Vibration feedback models for virtual environments , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[8]  S. Lacey,et al.  Vision and Touch: Multiple or Multisensory Representations of Objects? , 2007, Perception.

[9]  Lu Fang,et al.  Deep Learning for Surface Material Classification Using Haptic and Visual Information , 2015, IEEE Transactions on Multimedia.

[10]  Nawid Jamali,et al.  Majority Voting: Material Classification by Tactile Sensing Using Surface Texture , 2011, IEEE Transactions on Robotics.

[11]  Max Mintz,et al.  Refined methods for creating realistic haptic virtual textures from tool-mediated contact acceleration data , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[12]  Yoji Yamada,et al.  Psychophysical Dimensions of Tactile Perception of Textures , 2013, IEEE Transactions on Haptics.

[13]  Yasemin Vardar,et al.  Fingertip Interaction Metrics Correlate with Visual and Haptic Perception of Real Surfaces , 2019, 2019 IEEE World Haptics Conference (WHC).

[14]  R. Klatzky,et al.  Haptic perception: A tutorial , 2009, Attention, perception & psychophysics.

[15]  Eckehard G. Steinbach,et al.  Surface classification using acceleration signals recorded during human freehand movement , 2015, 2015 IEEE World Haptics Conference (WHC).

[16]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[18]  Peter A. Flach,et al.  Machine Learning - The Art and Science of Algorithms that Make Sense of Data , 2012 .

[19]  W. B. Tiest.  Tactual perception of material properties , 2010, Vision Research.

[20]  R. Klatzky,et al.  Feeling textures through a probe: Effects of probe and surface geometry and exploratory factors , 2003, Perception & psychophysics.

[21]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[22]  Katherine J. Kuchenbecker,et al.  Improving contact realism through event-based haptic feedback , 2006, IEEE Transactions on Visualization and Computer Graphics.

[23]  T. Hackett,et al.  Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[24]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.

[25]  M. Hollins,et al.  Pacinian representations of fine surface texture , 2005, Perception & psychophysics.

[26]  Seungmoon Choi,et al.  Data-driven thermal rendering: An initial study , 2018, 2018 IEEE Haptics Symposium (HAPTICS).

[27]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[28]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[29]  Heather Culbertson,et al.  Modeling and Rendering Realistic Textures from Unconstrained Tool-Surface Interactions , 2014, IEEE Transactions on Haptics.

[30]  J. Edward Colgate,et al.  High-bandwidth tribometry as a means of recording natural textures , 2017, 2017 IEEE World Haptics Conference (WHC).

[31]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[32]  Joseph M. Romano,et al.  Haptography: Capturing and Recreating the Rich Feel of Real Surfaces , 2009, ISRR.

[33]  Lu Fang,et al.  Preprocessing-free surface material classification using convolutional neural networks pretrained by sparse Autoencoder , 2015, 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP).

[34]  Gerald E. Loeb,et al.  Bayesian Exploration for Intelligent Identification of Textures , 2012, Front. Neurorobot..

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  Joseph M. Romano,et al.  Dimensional Reduction of High-Frequency Accelerations for Haptic Rendering , 2010, EuroHaptics.

[38]  Yon Visell,et al.  Learning constituent parts of touch stimuli from whole hand vibrations , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[39]  Heather Culbertson,et al.  One hundred data-driven haptic texture models and open-source methods for rendering on 3D objects , 2014, 2014 IEEE Haptics Symposium (HAPTICS).

[40]  Joseph M. Romano,et al.  Methods for robotic tool-mediated haptic surface recognition , 2014, 2014 IEEE Haptics Symposium (HAPTICS).

[41]  Gordon Cheng,et al.  Robust Tactile Descriptors for Discriminating Objects From Textural Properties via Artificial Robotic Skin , 2018, IEEE Transactions on Robotics.

[42]  Eckehard G. Steinbach,et al.  Multimodal Feature-Based Surface Material Classification , 2017, IEEE Transactions on Haptics.

[43]  A. Kappers,et al.  Haptic and visual perception of roughness. , 2007, Acta psychologica.

[44]  S. Lederman.  Tactile roughness of grooved surfaces: The touching process and effects of macro- and microsurface structure , 1974 .

[45]  Katherine J. Kuchenbecker,et al.  Handling Scan-time Parameters in Haptic Surface Classification , 2017, 2017 IEEE World Haptics Conference (WHC).

[46]  Muhammad Abdullah,et al.  Towards Universal Haptic Library: Library-Based Haptic Texture Assignment Using Image Texture and Perceptual Space , 2016, IEEE Transactions on Haptics.

[47]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[48]  Subhransu Maji,et al.  Deep filter banks for texture recognition and segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Takashi Yoshioka,et al.  Automatic filter design for synthesis of haptic textures from recorded acceleration data , 2010, 2010 IEEE International Conference on Robotics and Automation.

[50]  Eckehard G. Steinbach,et al.  Content-based surface material retrieval , 2017, 2017 IEEE World Haptics Conference (WHC).

[51]  Cagatay Basdogan,et al.  Effect of Waveform on Tactile Perception by Electrovibration Displayed on Touch Screens , 2017, IEEE Transactions on Haptics.

[52]  Seungmoon Choi,et al.  Geometry-based haptic texture modeling and rendering using photometric stereo , 2018, 2018 IEEE Haptics Symposium (HAPTICS).

[53]  Fabrizio Smeraldi,et al.  Comparative Evaluation of Hand-Crafted Image Descriptors vs. Off-the-Shelf CNN-Based Features for Colour Texture Classification under Ideal and Realistic Conditions , 2019, Applied Sciences.

[54]  Libor Ladanyi,et al.  Surface detection and recognition using infrared light , 2014, 2014 ELEKTRO.

[55]  Mohamad Eid,et al.  Measurement-Based Thermal Modeling Using Laser Thermography , 2018, IEEE Transactions on Instrumentation and Measurement.

[56]  Jivko Sinapov,et al.  Vibrotactile Recognition and Categorization of Surfaces by a Humanoid Robot , 2011, IEEE Transactions on Robotics.

[57]  D. Katz.  Der Aufbau der Tastwelt , 1925 .

[58]  Heather Culbertson,et al.  Importance of Matching Physical Friction, Hardness, and Texture in Creating Realistic Haptic Virtual Surfaces , 2017, IEEE Transactions on Haptics.

[59]  Jan Peters,et al.  Evaluation of tactile feature extraction for interactive object recognition , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[60]  Matti Pietikäinen,et al.  From BoW to CNN: Two Decades of Texture Representation for Texture Classification , 2018, International Journal of Computer Vision.

[61]  M. Hollins,et al.  The vibrations of texture , 2003, Somatosensory & motor research.

[62]  Trevor Darrell,et al.  Robotic learning of haptic adjectives through physical interaction , 2015, Robotics Auton. Syst..

[63]  Yoji Yamada,et al.  Wearable Finger Pad Sensor for Tactile Textures Using Propagated Deformation on a Side of a Finger: Assessment of Accuracy , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[64]  Yang Gao,et al.  Deep learning for tactile understanding from visual and haptic data , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).