A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem for model-based anomaly detection. We introduce a long short-term memory-based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution by introducing a progress-based varying prior. Our LSTM-VAE-based detector reports an anomaly when a reconstruction-based anomaly score is higher than a state-based threshold. For evaluations with 1555 robot-assisted feeding executions, including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve of 0.8710 than 5 other baseline detectors from the literature. We also show the variational autoencoding and state-based thresholding are effective in detecting anomalies from 17 raw sensory signals without significant feature engineering effort.

[1]  Francesco Piazza,et al.  Acoustic novelty detection with adversarial autoencoders , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[2]  Christian Osendorfer,et al.  Learning Stochastic Recurrent Networks , 2014, NIPS 2014.

[3]  William D. Smart,et al.  Robots for humanity: using assistive robotics to empower people with disabilities , 2013, IEEE Robotics & Automation Magazine.

[4]  Hokeun Kim,et al.  Multimodal anomaly detection for assistive robots , 2019, Auton. Robots.

[5]  Justin Bayer,et al.  Variational Inference for On-line Anomaly Detection in High-Dimensional Time Series , 2016, ArXiv.

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Xi Zhang,et al.  Integration of Data-Level Fusion Model and Kernel Methods for Degradation Modeling and Prognostic Analysis , 2018, IEEE Transactions on Reliability.

[8]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[9]  O. Ogorodnikova.  Methodology of safety for a human robot interaction designing stage , 2008, 2008 Conference on Human System Interactions.

[10]  Charles C. Kemp,et al.  Towards Assistive Feeding with a General-Purpose Mobile Manipulator , 2016, ArXiv.

[11]  Advait Jain,et al.  Reaching in clutter with whole-arm tactile sensing , 2013, Int. J. Robotics Res..

[12]  Tiffany L. Chen,et al.  Robots for Humanity : A Case Study in Assistive Mobile Manipulation , 2012 .

[13]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[14]  Chao Lan,et al.  Anomaly Detection , 2018, Encyclopedia of GIS.

[15]  Paul Levi,et al.  A Novel Framework for Anomaly Detection of Robot Behaviors , 2015, J. Intell. Robotic Syst..

[16]  Einoshin Suzuki,et al.  ACE: Anomaly Clustering Ensemble for Multi-perspective Anomaly Detection in Robot Behaviors , 2011, SDM.

[17]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[18]  Lovekesh Vig,et al.  Anomaly detection in ECG time signals via deep long short-term memory networks , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[19]  T.Y. Lin,et al.  Anomaly detection , 1994, Proceedings New Security Paradigms Workshop.

[20]  Lovekesh Vig,et al.  Long Short Term Memory Networks for Anomaly Detection in Time Series , 2015, ESANN.

[21]  Shiyu Zhou,et al.  Online Steady-State Detection for Process Control Using Multiple Change-Point Models and Particle Filters , 2016, IEEE Transactions on Automation Science and Engineering.

[22]  Stefan Schaal,et al.  Data-Driven Online Decision Making for Autonomous Manipulation , 2015, Robotics: Science and Systems.

[23]  Jun Morimoto,et al.  Estimating joint movements from observed EMG signals with multiple electrodes under sensor failure situations toward safe assistive robot control , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Charles C. Kemp,et al.  Multimodal execution monitoring for anomaly detection during robot manipulation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[25]  Siddhartha S. Srinivasa,et al.  Abort and retry in grasping , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[26]  Sungzoon Cho,et al.  Variational Autoencoder based Anomaly Detection using Reconstruction Probability , 2015 .

[27]  Yoshua Bengio,et al.  Denoising Criterion for Variational Auto-Encoding Framework , 2015, AAAI.

[28]  Timothy J. O'Shea,et al.  Recurrent Neural Radio Anomaly Detection , 2016, ArXiv.

[29]  Lovekesh Vig,et al.  LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection , 2016, ArXiv.

[30]  Yuki Suga,et al.  Multimodal integration learning of robot behavior using deep neural networks , 2014, Robotics Auton. Syst..

[31]  Christian Osendorfer,et al.  Model-free robot anomaly detection , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[33]  Maximilian Karl,et al.  Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data , 2016, ICLR.

[34]  Alberto Rodriguez,et al.  Failure detection in assembly: Force signature analysis , 2010, 2010 IEEE International Conference on Automation Science and Engineering.

[35]  Alexander Stoytchev,et al.  Learning to slide a magnetic card through a card reader , 2012, 2012 IEEE International Conference on Robotics and Automation.

[36]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.