Digital Assistance for Quality Assurance: Augmenting Workspaces Using Deep Learning for Tracking Near-Symmetrical Objects

We present a digital assistance approach for applied metrology on near-symmetrical objects. In manufacturing, systematically measuring products for quality assurance is often a manual task, where the primary challenge for the workers lies in accurately identifying positions to measure and correctly documenting these measurements. This paper focuses on a use-case, which involves metrology of small near-symmetrical objects, such as LEGO bricks. We aim to support this task through situated visual measurement guides. Aligning these guides poses a major challenge, since fine grained details, such as embossed logos, serve as the only feature by which to retrieve an object's unique orientation. We present a two-step approach, which consists of (1) locating and orienting the object based on its shape, and then (2) disambiguating the object's rotational symmetry based on small visual features. We apply and compare different deep learning approaches and discuss our guidance system in the context of our use case.

[1]  Wendy E. Mackay,et al.  Back to the real world , 1993, CACM.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Blair MacIntyre,et al.  An evaluation of graphical context when the graphics are outside of the task area , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[5]  Luis Perez,et al.  The Effectiveness of Data Augmentation in Image Classification using Deep Learning , 2017, ArXiv.

[6]  Ulrich Neumann,et al.  Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[7]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[8]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[9]  Steven K. Feiner,et al.  Augmented reality in the psychomotor phase of a procedural task , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[10]  Emiko Charbonneau,et al.  Interactive 3D Model Acquisition and Tracking of Building Block Structures , 2012, IEEE Transactions on Visualization and Computer Graphics.

[11]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[12]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Patrick Baudisch,et al.  Focus plus context screens: combining display technology with visualization techniques , 2001, UIST '01.

[14]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[16]  Steven K. Feiner,et al.  Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair , 2011, IEEE Transactions on Visualization and Computer Graphics.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Roman Rädle,et al.  PolarTrack: Optical Outside-In Device Tracking that Exploits Display Polarization , 2018, CHI.

[19]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[20]  Volker Paelke,et al.  Augmented reality in the smart factory: Supporting workers in an industry 4.0. environment , 2014, Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA).

[21]  S. Monsell,et al.  Costs of a predictible switch between simple cognitive tasks. , 1995 .

[22]  Tim Hauke Heibel,et al.  A mobile markerless AR system for maintenance and repair , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[23]  Quoc V. Le,et al.  Do Better ImageNet Models Transfer Better? , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Andrea Cavallaro,et al.  Multi-Camera Networks: Principles and Applications , 2009 .

[25]  Joost van de Weijer,et al.  Robust Estimation of Orientation for Texture Analysis , 2002 .

[26]  R. Shepard,et al.  Mental Rotation of Three-Dimensional Objects , 1971, Science.

[27]  Anind K. Dey,et al.  Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation , 2009, CHI.

[28]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[29]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[30]  L. Cooper.  Mental rotation of random two-dimensional shapes , 1975, Cognitive Psychology.

[31]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[32]  G. L. Cascella,et al.  Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations , 2018 .

[33]  Jörg Müller,et al.  Velt: A Framework for Multi RGB-D Camera Systems , 2018, ISS.

[34]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[35]  G. Michael Poor,et al.  "How Do I Line Up?": Reducing Mental Transformations to Improve Performance , 2011, HCI.

[36]  Woodrow Barfield,et al.  Evaluating the effectiveness of augmented reality displays for a manual assembly task , 1999, Virtual Reality.

[37]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[38]  Zhi Zhang,et al.  Bag of Tricks for Image Classification with Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Frank Biocca,et al.  Comparative effectiveness of augmented reality in object assembly , 2003, CHI '03.

[40]  Didier Stricker,et al.  Augmented reality for construction tasks: doorlock assembly , 1999 .

[41]  Hirokazu Kato,et al.  Handheld Guides in Inspection Tasks: Augmented Reality versus Picture , 2018, IEEE Transactions on Visualization and Computer Graphics.

[42]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Hamid Soltanian-Zadeh,et al.  Radon transform orientation estimation for rotation invariant texture analysis , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.