Bokeh Effect in Images on Objects Based on User Interest

Humans pay visual attention to those objects in the visual field that they are most interested in seeing. The Bokeh effect is a popular blurring effect in photography, where the object of interest is emphasized by blurring other objects. In this paper, we apply the principle of visual attention to the user's object of interest to post processing of photos taken using a smartphone. We simulate the Bokeh effect of blurring objects in the image except those that the user is interested. This adds a biologically inspired effect to the camera and gallery apps in the smartphone. We first define a hierarchy of user interests in different categories. We then create a user interest profile based on the user's demographics, apps and URLs. We build a user interest vector out of this hierarchy by using a word embedding model, and take the weighted average of the vectors of the words corresponding to the user interests. After this, we detect objects in the image and calculate the similarity of the detected objects with the user interest vector, returning a sorted list of objects the user is interested. The Bokeh effect is applied to the image to blur other objects, thus giving a realistic touch to the image. Finally, we conduct a user study to validate the effectiveness of the system.

[1]  Clément Viard,et al.  Image quality benchmark of computational bokeh , 2018 .

[2]  Dongwei Liu Computer Vision Meets Arts: Stereo Computational Photography , 2016 .

[3]  Luc Van Gool,et al.  The Interestingness of Images , 2013, 2013 IEEE International Conference on Computer Vision.

[4]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[5]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[6]  Ran Liu,et al.  Using Depth Mapping to realize Bokeh effect with a single camera Android device , 2016 .

[7]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  K. Hawk A Technical View of Bokeh , 1998 .

[9]  James Ze Wang,et al.  Training data collection system for a learning-based photographic aesthetic quality inference engine , 2010, ACM Multimedia.

[10]  Mounia Lalmas,et al.  On saliency, affect and focused attention , 2012, CHI.

[11]  Barbara Flueckiger Lifestyle, aesthetics and narrative in luxury domain advertising , 2009 .

[12]  Masashi Nishiyama,et al.  Aesthetic quality classification of photographs based on color harmony , 2011, CVPR 2011.

[13]  Shenghua Gao,et al.  Personalized Saliency and Its Prediction , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Joseph Redmon,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.