Augmented Reality Supported Modeling of Industrial Systems to Infer Software Configuration

This paper proposes and evaluates an approach for building models of installed industrial Cyber-Physical Systems using augmented reality on smartphones. It proposes a visual language for annotating devices, containers, flows of liquids and networking connections in augmented reality. Compared to related work, it provides a more lightweight and flexible approach for building 3D models of industrial systems. The models are further used to automatically infer software configuration of controllable industrial products. This addresses a common problem of error-prone and time-consuming configuration of industrial systems in the current practice. The proposed approach is evaluated in a study with 16 domain experts. The study participants are involved in creating a model of an industrial system for water treatment. Their comments show that the approach can enable a less error-prone configuration for more complex systems. Opportunities for improvement in usability and reflections on the potential of the approach are discussed.

[1]  Reimar Schumann,et al.  Virtual Commissioning Of Manufacturing Systems A Review And New Approaches For Simplification , 2010, ECMS.

[2]  Kleanthis Thramboulidis,et al.  A framework for MDE of IoT-based manufacturing cyber-physical systems , 2017, IOT.

[3]  Roland Rosen,et al.  About The Importance of Autonomy and Digital Twins for the Future of Manufacturing , 2015 .

[4]  Kaj Grønbæk,et al.  A visual programming approach based on domain ontologies for configuring industrial IoT installations , 2017, IOT.

[5]  Timothy I. Salsbury,et al.  A SURVEY OF CONTROL TECHNOLOGIES IN THE BUILDING AUTOMATION INDUSTRY , 2005 .

[6]  Simon Mayer,et al.  User interfaces for smart things -- A generative approach with semantic interaction descriptions , 2014, TCHI.

[7]  Jürgen Gausemeier,et al.  AR-Planning Tool - Designing Flexible Manufacturing Systems with Augmented Reality , 2002, EGVE.

[8]  Tovi Grossman,et al.  Ivy: Exploring Spatially Situated Visual Programming for Authoring and Understanding Intelligent Environments , 2017, Graphics Interface.

[9]  Reijo Sulonen,et al.  Towards a general ontology of configuration , 1998, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[10]  Gerd Kortuem,et al.  Market-based user innovation in the Internet of Things , 2010, 2010 Internet of Things (IOT).

[11]  Florian Michahelles,et al.  UberManufacturing: A Goal-Driven Collaborative Industrial Manufacturing Marketplace , 2016, IOT.

[12]  Franz Auinger,et al.  Interface driven domain-independent modeling architecture for “soft-commissioning” and “reality in the loop” , 1999, WSC '99.

[13]  Markus Funk,et al.  Antonius: a mobile search engine for the physical world , 2013, UbiComp.

[14]  Rik Van de Walle,et al.  Configuration of smart environments made simple: Combining visual modeling with semantic metadata and reasoning , 2014, 2014 International Conference on the Internet of Things (IOT).

[15]  Tobias Höllerer,et al.  Interpreting 2D gesture annotations in 3D augmented reality , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[16]  Nigel Shadbolt,et al.  Resource Description Framework (RDF) , 2009 .

[17]  Kaj Grønbæk,et al.  Building Models of Installations to Recommend Applications in IoT Software Ecosystems , 2016, 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud).

[18]  Iiro Harjunkoski,et al.  The impact of digitalization on the future of control and operations , 2017, Comput. Chem. Eng..

[19]  Hongbo Fu,et al.  Two‐Finger Gestures for 6DOF Manipulation of 3D Objects , 2012, Comput. Graph. Forum.

[20]  Morimasa Ogawa,et al.  The state of the art in chemical process control in Japan: Good practice and questionnaire survey , 2010 .

[21]  Ruben Mayer,et al.  The Fog Makes Sense: Enabling Social Sensing Services with Limited Internet Connectivity , 2017, SocialSens@CPSWeek.

[22]  George Vosselman,et al.  3D Reconstruction of Industrial Installations by Constrained Fitting of CAD Models to Images , 2000, DAGM-Symposium.

[23]  Christian Bonnet,et al.  LOV4IoT: A Second Life for Ontology-Based Domain Knowledge to Build Semantic Web of Things Applications , 2016, 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud).

[24]  Alexander Fay,et al.  Automatic derivation of qualitative plant simulation models from legacy piping and instrumentation diagrams , 2016, Comput. Chem. Eng..

[25]  Matthias Nießner,et al.  SemanticPaint , 2015, ACM Trans. Graph..

[26]  M. Porter,et al.  How Smart, Connected Products Are Transforming Companies , 2015 .

[27]  Stefan Rennick Egglestone,et al.  Digital plumbing: the mundane work of deploying UbiComp in the home , 2010, Personal and Ubiquitous Computing.

[28]  Jan Bosch.  Software Product Line Engineering , 2013, Systems and Software Variability Management.

[29]  Pattie Maes,et al.  Smarter objects: using AR technology to program physical objects and their interactions , 2013, CHI Extended Abstracts.

[30]  Andrew Y. C. Nee,et al.  Augmented reality applications in manufacturing: a survey , 2008 .

[31]  A. Schmidt,et al.  motionEAP : An Overview of 4 Years of Combining Industrial Assembly with Augmented Reality for Industry 4 . 0 , 2016 .

[32]  Changmin Kim,et al.  Skeleton-based 3D reconstruction of as-built pipelines from laser-scan data , 2013 .

[33]  Changmin Kim,et al.  3D reconstruction of as-built industrial instrumentation models from laser-scan data and a 3D CAD database based on prior knowledge , 2015 .

[34]  Kris Luyten,et al.  Pervasive maps: Explore and interact with pervasive environments , 2010, 2010 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[35]  Kamin Whitehouse,et al.  Smart Blueprints: Automatically Generated Maps of Homes and the Devices Within Them , 2012, Pervasive.

[36]  Barry Brumitt,et al.  Ubiquitous computing and the role of geometry , 2000, IEEE Wirel. Commun..

[37]  S. Jiang,et al.  An AR-based hybrid approach for facility layout planning and evaluation for existing shop floors , 2014, The International Journal of Advanced Manufacturing Technology.

[38]  Jan Bosch,et al.  Architecture for embedded open software ecosystems , 2014, J. Syst. Softw..

[39]  Mani Srivastava,et al.  Brick: Towards a Unified Metadata Schema For Buildings , 2016, BuildSys@SenSys.