Gibson Env V2: Embodied Simulation Environments for Interactive Navigation

Autonomous navigation is one of the most crucial tasks for mobile agents. The goal is to have the mobile agent reach any location in the environment in a safe and robust manner. Traditionally, robot navigation (including obstacle avoidance) has been addressed with analytical model-based solutions using signals from Lidars or depth sensors [5, 6]. Recently, learning based visual navigation methods have gained popularity because 1) they can perform navigation without accurate localization or metric maps [18, 1], 2) they do not require expensive Lidars or depth sensors [11, 14], 3) they can generalize robustly in previously unseen environments [24, 15]. Despite these benefits, learning-based approaches usually require a large amount of data. Collecting the data through interactions with the real world could be dangerous, costly and time-consuming. A solution to this challenge of learning-based navigation is to learn in simulated environments. In simulation, the agent can collect experiences safely and efficiently: usually one or two orders of magnitude faster than real-time. However, despite recent advances in simulation for robotics [19, 24, 20, 21, 12, 10, 7, 3], it is still less than straightforward to transfer directly what is learned in simulation to the real world. The reason for this is the so-called sim2real gap: the (more or less subtle) differences between the simulated and real environments, for example due to the different spatial arrangement of objects or the disparity between real and simulated sensor signals. Many learning-based approaches rely on simulation for fast policy learning. And different simulation-to-real transfer strategies including photorealistic rendering[16, 4], domain randomization[17] and domain adaptation[2, 23] were proposed. In particular, the Gibson Env[23] is a simulation environment that does photorealistic rendering and additionally provides a pixel-level domain adaptation mechanism to aid with the commonly raised concern of sim-to-real transfer. In Gibson Env the spatial arrangement of objects is realistic because the models of the environments are obtained from the real world. Additionally, the gap between simulated and real visual sensors is bridged through a novel neural network, the Goggles. Thanks to these properties, Gibson Env has demonstrated great sim-to-real transfer performance [9, 13]. However, Gibson Env presents two main limitations that hinder its use for learning-based navigation approaches: 1) the rendering is relatively slow (40-100fps) for large-scale training, which partially defeats the purpose of training in simulation, and 2) the interaction between the agent and the environment is limited only to a planar motion on the floor, while navigation in many real-world scenarios involves more intricate forms of interaction, such as opening doors and pushing away objects that are in the way.