MyFixit: An Annotated Dataset, Annotation Tool, and Baseline Methods for Information Extraction from Repair Manuals

Text instructions are among the most widely used media for learning and teaching. Hence, to create assistance systems that are capable of supporting humans autonomously in new tasks, it would be immensely productive, if machines were enabled to extract task knowledge from such text instructions. In this paper, we, therefore, focus on information extraction (IE) from the instructional text in repair manuals. This brings with it the multiple challenges of information extraction from the situated and technical language in relatively long and often complex instructions. To tackle these challenges, we introduce a semi-structured dataset of repair manuals. The dataset is annotated in a large category of devices, with information that we consider most valuable for an automated repair assistant, including the required tools and the disassembled parts at each step of the repair progress. We then propose methods that can serve as baselines for this IE task: an unsupervised method based on a bags-of-n-grams similarity for extracting the needed tools in each repair step, and a deep-learning-based sequence labeling model for extracting the identity of disassembled parts. These baseline methods are integrated into a semi-automatic web-based annotator application that is also available along with the dataset.

[1]  Barbara G Deutsch,et al.  The Structure of Task Oriented Dialogs , 1974 .

[2]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[3]  Roland Vollgraf,et al.  Pooled Contextualized Embeddings for Named Entity Recognition , 2019, NAACL.

[4]  Roland Vollgraf,et al.  FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP , 2019, NAACL.

[5]  Ziqi Zhang,et al.  Automatically Extracting Procedural Knowledge from Instructional Texts using Natural Language Processing , 2012, LREC.

[6]  David Whitney,et al.  Interpreting multimodal referring expressions in real time , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Mikhail Khodak,et al.  A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs , 2018, ICLR.

[8]  Yoko Yamakata,et al.  A method for extracting major workflow composed of ingredients, tools, and actions from cooking procedural text , 2016, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[9]  Kalina Bontcheva,et al.  Developing Language Processing Components with GATE (a User Guide) , 2003 .

[10]  Dejan Pangercic,et al.  Web-enabled Robots -- Robots that Use the Web as an Information Resource , 2011, ICRA 2011.

[11]  Shinsuke Mori,et al.  A Framework for Procedural Text Understanding , 2015, IWPT.

[12]  Grzegorz Kondrak,et al.  N-Gram Similarity and Distance , 2005, SPIRE.

[13]  Thomas G. Dietterich,et al.  Learning Scripts as Hidden Markov Models , 2014, AAAI.

[14]  Sampo Pyysalo,et al.  brat: a Web-based Tool for NLP-Assisted Text Annotation , 2012, EACL.

[15]  Herbert H. Clark,et al.  Coordinating beliefs in conversation , 1992 .

[16]  Yoko Yamakata,et al.  Flow Graph Corpus from Recipe Texts , 2014, LREC.

[17]  Andrei Z. Broder,et al.  On the resemblance and containment of documents , 1997, Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171).

[18]  Gerhard Weikum,et al.  Distilling Task Knowledge from How-To Communities , 2017, WWW.

[19]  Stefanie Tellex,et al.  Interpreting and Executing Recipes with a Cooking Robot , 2012, ISER.

[20]  Raymond J. Mooney,et al.  Learning Statistical Scripts with LSTM Recurrent Neural Networks , 2016, AAAI.

[21]  Kalina Bontcheva,et al.  Developing Language Processing Components with GATE Version 5 (a User Guide) , 2010 .

[22]  Earl J. Wagner,et al.  Cooking with Semantics , 2014, ACL 2014.

[23]  Siobhan Chapman.  Logic and Conversation , 2005 .

[24]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[25]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[26]  Mark Steedman,et al.  Extracting common sense knowledge from text for robot planning , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Eduardo Salas,et al.  Situation Awareness in Team Performance: Implications for Measurement and Training , 1995, Hum. Factors.