Photo image classification using pre-trained deep network for density-based spatiotemporal analysis system

Recently, during natural disasters like earthquakes, typhoons, flood, and heavy snowfall, messages and photos describing the situations faces by people are actively posted on social media sites. Therefore, the development of an analysis system using the data on social media sites to enhance situation awareness in the real world is an important research topic. In our previous work, we developed a density-based spatiotemporal analysis system to enhance situation awareness during emergency situations. The system could identify areas related to an observed emergency topic using tweet classifier, spatiotemporal clustering, and photo image classifier using the Bag-of-Features (BoF) model. In this paper, we propose a novel density-based spatiotemporal analysis system with a photo image classifier that used a pretrained deep network. The pre-trained deep network is integrated into the conventional photo image classifier instead of the BoF model. The proposed system can enhance situation awareness compared to our previous system by accurately classifying the photo images related to an observed emergency topic. To evaluate the proposed system, we used actual photo images attached to tweets related to “heavy rain” in Japan. The experimental results showed that the proposed system can classify photo images related to “heavy rain” more sensitively than our previous system.