Supplementary Material for “Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding”

In the proposed method, we approximate the reference SOS feature of an object token by using prior information about objects in the training data. For instance, for the ‘chair’ object, we collected the widths and heights of the detected bounding boxes as shown in Figure 1. Figure 2 shows two representative values: median and mean for each distribution. We choose the median values, which minimizes the L1 error, to represent the reference bounding box of each object. To generate rotation-invariant SOS features, we convert the four vertices of the bounding box detected from the front view image of size 640 × 480 to the vertices of a bounding box detected from the panoramic view image of size 2048 × 512 using coordinate transformations. To simplify the implementation, we assume that the converted bounding box has a rectangle shape with the vertices transformed into coordinates in a panoramic view. The reference SOS feature is calculated as the logarithmic magnitude of the Fourier transform of the panoramic mask with mean pooling on the vertical spectral axis. Considering that the shift in the spatial-domain only affects the phase of the Fourier transform, the location of a reference bounding box does not matter.