TA-CNN: Two-way attention models in deep convolutional neural network for plant recognition

Abstract Automatic plant recognition using AI is a challenging problem. In addition to the recognition of the plant specimen, we also want to recognize the plant type in its actual living environment, which is more difficult because of the background noise. In this paper, we propose a novel method, referred to as the two-way attention model using the deep convolutional neural network. As the name implies, it has two ways of attentions: the first attention way is the family first attention, which is based on the standard plant taxonomy and aims to recognize the plant’s family. Specifically, we create plant family labels as another objective of the learning under the multi-task learning framework. To deal with conflicting prediction of family and species labels, we propose an implicit tree model with a dedicated loss function to maintain the correspondence between family and species labels. The second attention way is the max-sum attention, which focuses on the discriminative features of the input image by finding the max-sum part of the fully convolutional network heat map. Because these two ways of attention are compatible, we combine both discriminative feature learning and part based attention. The experiments on four challenging datasets (i.e. Malayakew, ICL, Flowers 102 and CFH plant) confirm the effectiveness of our method – the recognition accuracy over those four dataset reaches 99.8%, 99.9%, 97.2% and 79.5% respectively.