Development and Deployment of a Large-Scale Flower Recognition Mobile App

Today’s major image search engines, which focus on searchby-image-content, work by matching and retrieving of images that are already available on the web. With the proliferation of user generated content, especially from mobile devices, there is a need to develop applications which are more content-aware, i.e. can understand and recognize what is in the image, and which can handle the deteriorated quality inherent to user images uploaded from mobile devices. In this paper we describe a mobile phone application intended to automatically recognize flower images taken by users and classify them into a set of flower categories. The app is served by a web-scale image search engine and relies on computer vision recognition technology. The mobile phone app is available free to users, and as a web interface. We share experiences about designing, developing, and deploying such a system in practice. We discuss practical aspects of applying the object recognition technology, including runtime, scalability, and data collection which is crucial for such data-driven application. More specifically, we describe a large-scale dataset which was collected in a number of stages to be as close as possible to the data distribution the actual users of the app will encounter. We further describe our strategy for collecting user feedback, which we view as an opportunity to improve the server-side algorithms and datasets. We envision that these issues will be encountered in similar applications, e.g. for recognition of plants, mushrooms, or bird species, all of which have practical importance for users, and we hope this work will be also useful for developing other types of applications. To our knowledge, this is the first mobile app which can automatically recognize as many as 578 species of flowers, and which is available for free to users. The flower dataset that serves the application, specifically collected for this recognition task, is the first and largest in its scale and scope.

[1]  Shih-Fu Chang,et al.  Detection of non-identical duplicate consumer photographs , 2003, Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint.

[2]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[3]  Yihong Gong,et al.  Locality-constrained Linear Coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  C. V. Jawahar,et al.  The truth about cats and dogs , 2011, 2011 International Conference on Computer Vision.

[5]  Andrew Zisserman,et al.  BiCoS: A Bi-level co-segmentation method for image classification , 2011, 2011 International Conference on Computer Vision.

[6]  Changhu Wang,et al.  MindFinder: image search by interactive sketching and tagging , 2010, WWW '10.

[7]  Garrison W. Cottrell,et al.  Robust classification of objects, faces, and flowers using natural image statistics , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Pietro Perona,et al.  Caltech-UCSD Birds 200 , 2010 .

[9]  Ming Yang,et al.  Large-scale image classification: Fast feature extraction and SVM training , 2011, CVPR 2011.

[10]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Pietro Perona,et al.  Visual Recognition with Humans in the Loop , 2010, ECCV.

[12]  Satoshi Ito,et al.  Object Classification Using Heterogeneous Co-occurrence Features , 2010, ECCV.

[13]  Hao Xu,et al.  Interactive image search by 2D semantic map , 2010, WWW '10.

[14]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[15]  Nozha Boujemaa,et al.  The CLEF 2011 Plant Images Classification Task , 2011, CLEF.

[16]  Larry S. Davis,et al.  Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance , 2011, 2011 International Conference on Computer Vision.

[17]  W. John Kress,et al.  Leafsnap: A Computer Vision System for Automatic Plant Species Identification , 2012, ECCV.

[18]  Yan Ke,et al.  Efficient Near-duplicate Detection and Sub-image Retrieval , 2004 .