Exploring confusing scene classes for the places dataset: Insights and solutions

Scene classification is more challenging than object classification due to higher ambiguity in scene labels. In this work, we propose to use the filter weights at the last stage of a CNN model trained by the Places dataset, which is also known as the "scene anchor vector (SAV)", to explain the source of confusions. An SAV points to a cluster of images. If two anchor vectors have a smaller angle, we see overlapping image clusters, leading to a set of confusing classes. To overcome it, we propose to merge images associated with confusing anchor vectors into a confusion set and split the set in an unsupervised fashion to create multiple subsets. It is called the "automatic subset clustering (ASC)" process. Each of these subsets contains scene images of strong visual similarity. After the ASC process, we train a random forest (RF) classifier for each confusion subset to allow better scene classification. The ASC/RF scheme can be added on top of any existing scene-classification CNN as a post-processing module with little extra training effort. It is shown by extensive experimental results that, for a given baseline CNN, the ASC/RF scheme can offer a significant performance gain.