Media-rich streaming for remote simulation and training

We present a new approach for the real-time control and operation of media-rich workstations and simulations over remote networked computers. This approach allows a user to interact with a remote computer using a standard keyboard, mouse, and joystick at a high level of interactivity while using significantly less bandwidth than existing remote desktop applications. Existing remote desktop solutions that use no compression or single frame image compression are not able to update full-screen animation and video at an interactive rate over bandwidth constrained networks. The use of video compression allows high frame update rates with a low bandwidth usage. This allows a wide range of new media-rich telecommuting applications such as 3D graphical simulation and training. Simulation and training operations are often performed at dedicated facilities. There can be many reasons restricting these operations to a specific site, such as the use of expensive computers and the need for security to protect sensitive data used in the simulations. These geographical restrictions can have significant ramifications. Requiring training to be performed on site can both restrict the availability of such training as well as lead to high transportation and time costs and for users. By developing tools which allow remote operation of simulations, a broader deployment of training can be achieved, allowing more users to train and at a lower cost. Using video compression to stream live desktop visuals can leverage the computational facilities at simulation sites while requiring significantly less powerful computers to receive and display the video streams to end users. Furthermore, simulation data and other sensitive information can be stored on secure computers while only the visual output of the simulation is sent to the remote user. In this paper we present an implementation intended for the training of control console operators which allows remote visualization and interaction with a simulation at HD resolution over a 1.5 Mbps T-1 data line. This allows users to train while on active deployment by using a dummy console which is able to receive and display the video stream while sending user input events back to the simulation computer. The dummy console costs significantly less than an actual simulation console and can be deployed to virtually anywhere in the world with a satisfactory internet connection. Furthermore, we demonstrate the option of using virtual and augmented reality mixed with the streamed simulation content. This allows for training when console workspaces are either unavailable or can not be taken offline from their active use. We evaluate the performance of this system on a variety of mock-simulation applications and measure that the latency of the video streaming is less than 150 ms, which includes video encoding on the host computer and decoding and display on the user's computer. The network transportation time for long distance network communication adds additional latency. Including the time for user input to be sent, we demonstrate that the total end-to-end latency for a cross country simulation is 250 ms. With this performance, interactive remote training and simulation becomes possible.