Generating image descriptions using capsule network

Abstract Using Capsule network to generate natural language descriptions of images with different orientation, shapes, sizes and regions is a novel technique to address image to text problem. In this paper we are going to present a model that generates image descriptions using convolutional neural network and capsule network. First, the model will be trained on MNIST dataset for testing the accuracy of identifying the images with different orientation using capsule network and after that we used Flickr8K [3] and Flickr30K [4, 5] datasets over CNN and bidirectional recurrent neural network to generate text descriptions. We have added an additional steps to change the orientation of these images by rotating them to certain angles which will help in creating a new set of training samples for images with different orientations. Second, we analysed the performance of both the system separately and combined, calculate the aggregated result while taking in account the complexity of image descriptor and pose content of images. We found that this idea outperform significantly on describing images with different orientations and scalable to adapt new images apart from the training samples without predefined set of guidelines.