An Approach of Text-Based and Image-Based Multi-modal Search for Online Shopping

Nowadays, more and more people prefer online shopping to physical store shopping for its convenience, cheapness and timesaving. Customers visit some commercial shopping websites, and select their favorite commodities by accessing links or retrieving by search box. However, in our real life, most online shopping websites provide a simple and single text retrieval method only, to some extent it’s difficult for customers to submit query and retrieve satisfactory results. In this paper, a multi-modal search approach combining text-based and image-based search techniques is presented. Besides text search, a two-stage image search approach is proposed, which utilizes basic features consisting of color and textural features to filter mismatching images in first stage, and further uses SIFT features for accurate search in second stage. Moreover, a prototype system has been developed for multi-modal search on online shopping websites. By submitting some words, phrases, images or their combination, customers can search out what they want. The experiments compared with traditional algorithms based on single visual feature validate that our approach and multi-modal search prototype system are effective, and the retrieval results can satisfy customers’ requirements well for online shopping.