Augmented Reality Shopping System Through Image Search and Virtual Shop Generation

In this paper, we introduce an intelligent augmented reality shopping system that supports image search and 3D virtual shop generation. By capturing an image of the product that the user is looking at, and using image search, we can recommend similar products to the user. Based on the spatial understanding techniques, we developed a 3D virtual shop interface that would be automatically generated in the user’s physical room, where products are arranged around the room at appropriate locations. With the virtual shop display, users would be able to interact with the products’ life-size virtual models by full-hand manipulation. Our system provides more natural interactions such as grabbing and touching. We also carried out an evaluation study for using our system, the participants’ feedback shows that our system provides a better shopping experience with its intuitive search process, immersive shop display, and natural interactions.

[1]  Aron M. Levin,et al.  A MULTI-ATTRIBUTE ANALYSIS OF PREFERENCES FOR ONLINE AND OFFLINE SHOPPING: DIFFERENCES ACROSS PRODUCTS, CONSUMERS, AND SHOPPING STAGES , 2005 .

[2]  Avikarsha Mandal,et al.  Possible applications of the LEAP motion controller for more interactive simulated experiments in augmented or virtual reality , 2016, Optical Engineering + Applications.

[3]  Hongbo Fu,et al.  Two‐Finger Gestures for 6DOF Manipulation of 3D Objects , 2012, Comput. Graph. Forum.

[4]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[5]  Francesco Coccia,et al.  Virtual, augmented e mixed reality: analisi del mercato dei visori e dell’impatto sul segmento consumer e business: il caso Microsoft Hololens , 2019 .

[6]  Tamio Tanikawa,et al.  Development of micro manipulation system with two-finger micro hand , 1996, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96.

[7]  Yuzhu Lu,et al.  Augmented Reality E-Commerce Assistant System: Trying While Shopping , 2007, HCI.

[8]  Jean-François Lalonde,et al.  Real-Time High Resolution 3D Data on the HoloLens , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[9]  Qing He,et al.  From Online Behaviors to Offline Retailing , 2016, KDD.

[10]  Hao Dou,et al.  Show Something: Intelligent Shopping Assistant Supporting Quick Scene Understanding and Immersive Preview , 2019, HCI.

[11]  Angeline C. Scheinbaum,et al.  Beyond Buying: Motivations Behind Consumers’ Online Shopping Cart Use , 2009 .

[12]  Yiming Zhao,et al.  A 3D virtual shopping mall that has the intelligent virtual purchasing guider and cooperative purchasing functionalities , 2004, 8th International Conference on Computer Supported Cooperative Work in Design.

[13]  Jean-Rémy Chardonnet,et al.  Free Hand-Based 3D Interaction in Optical See-Through Augmented Reality Using Leap Motion , 2018 .

[14]  Siam Charoenseang,et al.  Intuitive 3D Model Prototyping with Leap Motion and Microsoft HoloLens , 2018, HCI.