Sentiment-based sub-event segmentation and key photo selection

Abstract The number of people collecting photos has surged owing to social media sharing and cloud services in recent years. A typical approach to summarize a photo collection is dividing it into events and selecting key photos from each event. Despite the fact that a certain event comprises several sub-events, few studies have proposed sub-event segmentation. Therefore, we propose the sentiment analysis-based photo summarization (SAPS) method, which automatically summarizes personal photo collections by utilizing metadata and visual sentiment features. For this purpose, we first cluster events using metadata of photos and then calculate the novelty scores to determine the sub-event boundaries. Next, we summarize the photo collections using a ranking algorithm that measures sentiment, emotion, and aesthetics. We evaluate the proposed method by applying it to the photo collections of six participants comprising of 5,480 photos in total. We observe that our sub-event segmentation based on sentiment features outperforms the existing baseline methods. Furthermore, the proposed method is also more effective in finding sub-event boundaries and key photos, because it focuses on detailed sentiment features instead of general content features.

[1]  Na Li,et al.  Affective image classification by jointly using interpretable art features and semantic annotations , 2019, J. Vis. Commun. Image Represent..

[2]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[3]  Tao Mei,et al.  Probabilistic Multimodality Fusion for Event based Home Photo Clustering , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[4]  Jonathan Foote,et al.  Automatic audio segmentation using a measure of audio novelty , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[5]  Claudia Niederée,et al.  To Keep or not to Keep: An Expectation-oriented Photo Selection Method for Personal Photo Collections , 2015, ICMR.

[6]  Liyan Zhang,et al.  Sub-event recognition and summarization for structured scenario photos , 2016, Multimedia Tools and Applications.

[7]  Andreas Paepcke,et al.  Time as essence for photo browsing through personal digital libraries , 2002, JCDL '02.

[8]  Jun-Hyuk Kim,et al.  Travel photo album summarization based on aesthetic quality, interestingness, and memorableness , 2016, 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[9]  Claudia Niederée,et al.  Investigating human behaviors in selecting personal photos to preserve memories , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[10]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yukun Li,et al.  Visualization of Photo Album: Selecting a Representative Photo of a Specific Event , 2019, DASFAA Workshops.

[12]  Antoine Pigeau.  Life Gallery: event detection in a personal media collection , 2016, Multimedia Tools and Applications.

[13]  Petia Radeva,et al.  SR-clustering: Semantic regularized clustering for egocentric photo streams segmentation , 2015, Comput. Vis. Image Underst..

[14]  Donald W. Bouldin,et al.  A Cluster Separation Measure , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Mary Czerwinski,et al.  PhotoTOC: automatic clustering for browsing personal photographs , 2003, Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint.

[16]  Matias Valdenegro-Toro,et al.  Real-time Convolutional Neural Networks for emotion and gender classification , 2017, ESANN.

[17]  Chong Tan,et al.  Research on Sentiment Classification of Active Scene Images Based on DNN , 2019, 2019 International Conference on Virtual Reality and Intelligent Systems (ICVRIS).

[18]  Yuzhen Niu,et al.  Rule of Thirds Detection from Photograph , 2011, 2011 IEEE International Symposium on Multimedia.

[19]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Petia Radeva,et al.  Smartphone picture organization: A hierarchical approach , 2018, Comput. Vis. Image Underst..

[21]  Rongrong Ji,et al.  SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content , 2013, ACM Multimedia.

[22]  Min-Yen Kan,et al.  Hidden Markov Model for Event Photo Stream Segmentation , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[23]  Jiebo Luo,et al.  Joint Visual-Textual Sentiment Analysis with Deep Neural Networks , 2015, ACM Multimedia.

[24]  Nadjia Benblidia,et al.  Event recognition in photo albums using probabilistic graphical models and feature relevance , 2016, J. Vis. Commun. Image Represent..

[25]  Hanghang Tong,et al.  Blur detection for digital images using wavelet transform , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[26]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[27]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Joo-Hwee Lim,et al.  Predicting Visual Context for Unsupervised Event Segmentation in Continuous Photo-streams , 2018, ACM Multimedia.

[29]  Sudeep Sarkar,et al.  A Perceptual Prediction Framework for Self Supervised Event Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Jonice Oliveira,et al.  Subevents detection through topic modeling in social media posts , 2019, Future Gener. Comput. Syst..

[31]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[32]  Naila Murray,et al.  AVA: A large-scale database for aesthetic visual analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Tania Pouli,et al.  Context in Photo Albums , 2019, ACM Trans. Appl. Percept..

[34]  Stefan Winkler,et al.  A Probabilistic Approach to People-Centric Photo Selection and Sequencing , 2017, IEEE Transactions on Multimedia.

[35]  Steffen Staab,et al.  Smart photo selection: interpret gaze as personal interest , 2014, CHI.

[36]  Nicola Conci,et al.  How Deep Features Have Improved Event Recognition in Multimedia , 2019, ACM Trans. Multim. Comput. Commun. Appl..

[37]  Xinmei Tian,et al.  Multi-modal and multi-scale photo collection summarization , 2015, Multimedia Tools and Applications.

[38]  Guohui Tian,et al.  A novel scene classification model combining ResNet based transfer learning and data augmentation with a filter , 2019, Neurocomputing.