Jarvis for Aeroengine Analytics: A Speech Enhanced Virtual Reality Demonstrator Based on Mining Knowledge Databases

In this paper, we present a Virtual Reality (VR) based environment where the engineer interacts with incoming data from a fleet of aeroengines. This data takes the form of 3D computer-aided design (CAD) engine models coupled with characteristic plots for the subsystems of each engine. Both the plots and models can be interacted with and manipulated using speech or gestural input. The characteristic data is ported to a knowledge-based system underpinned by a knowledge-graph storing complex domain knowledge. This permits the system to respond to queries about the current state and health of each aeroengine asset. Responses to these questions require some degree of analysis, which is handled by a semantic knowledge representation layer managing information on aeroengine subsystems. This paper represents a significant step forward for aeroengine analysis in a bespoke VR environment and brings us a step closer to a Jarvis-like system for aeroengine analytics.

[1]  Per Ola Kristensson,et al.  IPCP: Immersive Parallel Coordinates Plots for Engineering Design Processes , 2020, AIAA Scitech 2020 Forum.

[2]  John R. Wright,et al.  Immersive environment technologies for planetary exploration , 2001, Proceedings IEEE Virtual Reality 2001.

[3]  Ivan Herman,et al.  Graph Visualization and Navigation in Information Visualization: A Survey , 2000, IEEE Trans. Vis. Comput. Graph..

[4]  R. Bonney,et al.  Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy , 2009 .

[5]  Pierre Le Sidaner,et al.  Defining and cataloging exoplanets: the exoplanet.eu database , 2011, 1106.0586.

[6]  John L. Schmalzel,et al.  Visualization of multiple sensor measurements in a VR environment for integrated systems health management in rocket engine tests , 2009, 2009 IEEE Sensors Applications Symposium.

[7]  John L. Schmalzel,et al.  Acquisition, interfacing and analysis of sensor measurements in a VR environment for integrated systems health management in rocket engine tests , 2009, 2009 IEEE Sensors Applications Symposium.

[8]  D. W. Mizell.  Virtual reality and augmented reality in aircraft design and manufacturing , 1994, Proceedings of WESCON '94.

[9]  G.M. Stump,et al.  Trade space exploration of satellite datasets using a design by shopping paradigm , 2004, 2004 IEEE Aerospace Conference Proceedings (IEEE Cat. No.04TH8720).

[10]  Mikel Sagardia,et al.  An interactive virtual reality system for on-orbit servicing , 2013, 2013 IEEE Virtual Reality (VR).

[11]  S. Kavitha,et al.  Development of automatic speech recognition system for voice activated Ground Control system , 2015, 2015 International Conference on Trends in Automation, Communications and Computing Technology (I-TACT-15).

[12]  C.S. Hartley,et al.  Use of heads-up displays, speech recognition, and speech synthesis in controlling a remotely piloted space vehicle , 1988, IEEE Aerospace and Electronic Systems Magazine.

[13]  Robert J. Stone,et al.  Evolution of aerospace simulation: From immersive Virtual Reality to serious games , 2011, Proceedings of 5th International Conference on Recent Advances in Space Technologies - RAST2011.

[14]  An Update to “ A Review of Space Environment Implications of CubeSat Traffic ” , 2015 .

[15]  Pascal Guitton,et al.  3D collaborative interaction for aerospace industry , 2016, 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE).

[16]  Robin Wolff,et al.  Collaborative telepresence workspaces for space operation and science , 2015, 2015 IEEE Virtual Reality (VR).

[17]  Alfred Inselberg,et al.  Parallel coordinates: a tool for visualizing multi-dimensional geometry , 1990, Proceedings of the First IEEE Conference on Visualization: Visualization `90.

[18]  Per Ola Kristensson,et al.  Digital Twin Assessments in Virtual Reality: An Explorational Study with Aeroengines , 2020, 2020 IEEE Aerospace Conference.

[19]  Steffen Staab,et al.  Knowledge graphs , 2021, Commun. ACM.

[20]  Dieter Kranzlmuller,et al.  Perspectives for using virtual reality to extend visual data mining in information visualization , 2016, 2016 IEEE Aerospace Conference.

[21]  L. G. Allred,et al.  Achieving a hands-free computer interface using voice recognition and speech synthesis [for Windows-based ATE] , 2000 .

[22]  Pj Clarkson,et al.  Good design practice for medical devices and equipment - requirements capture , 2002 .

[23]  Diego Borro,et al.  Description of a haptic system for virtual maintainability in aeronautics , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  Ken Museth,et al.  Semi-immersive space mission design and visualization: case study of the "Terrestrial Planet Finder" mission , 2001, Proceedings Visualization, 2001. VIS '01..

[25]  Alvin Jude,et al.  Grasp, Grab or Pinch? Identifying User Preference for In-Air Gestural Manipulation , 2016, SUI.

[26]  Shigeru Obayashi,et al.  Data Mining for Aerodynamic Design Space , 2005, J. Aerosp. Comput. Inf. Commun..

[27]  Oliver Ohneiser,et al.  Reducing controller workload with automatic speech recognition , 2016, 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC).

[28]  John M. Reising,et al.  The simultaneous use of three machine speech recognition systems to increase recognition accuracy , 1994, Proceedings of National Aerospace and Electronics Conference (NAECON'94).

[29]  J. P. Hale,et al.  Applied virtual reality in aerospace design , 1994, Proceedings of WESCON '94.

[30]  Per Ola Kristensson,et al.  PhotoTwinVR: An Immersive System for Manipulation, Inspection and Dimension Measurements of the 3D Photogrammetric Models of Real-Life Structures in Virtual Reality , 2019, ArXiv.

[31]  Yves Raimond,et al.  RDF 1.1 Primer , 2014 .

[32]  Nick Pawlowski,et al.  Rasa: Open Source Language Understanding and Dialogue Management , 2017, ArXiv.