Semantically Smoothed Refinement for Everyday Concept Indexing

Instead of occurring independently, semantic concepts pairs tend to co-occur within a single image and it is intuitive that concept detection accuracy for visual concepts can be enhanced if concept correlation can be leveraged in some way. In everyday concept detection for visual lifelogging using wearable cameras to automatically record everyday activities, the captured images usually have a diversity of concepts which challenges the performance of concept detection. In this paper a semantically smoothed refinement algorithm is proposed using concept correlations which exploit topic-related concept relationships, modeled externally in a user experiment rather than extracted from training data. Results for initial concept detection are factorized based on semantic smoothness and adjusted in compliance with the extracted concept correlations. Refinement performance is demonstrated in experiments to show the effectiveness of our algorithm and the extracted correlations.

[1]  Alan F. Smeaton,et al.  Constructing a SenseCam visual diary as a media process , 2008, Multimedia Systems.

[2]  George Lakoff,et al.  Women, Fire, and Dangerous Things , 1987 .

[3]  Tao Mei,et al.  Correlative multi-label video annotation , 2007, ACM Multimedia.

[4]  Qiang Yang,et al.  One-Class Collaborative Filtering , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[5]  Djoerd Hiemstra,et al.  Simulating the future of concept-based video retrieval under improved detector performance , 2011, Multimedia Tools and Applications.

[6]  Alan F. Smeaton,et al.  LifeLogging: Personal Big Data , 2014, Found. Trends Inf. Retr..

[7]  Jianping Fan,et al.  Correlative multi-label multi-instance image annotation , 2011, 2011 International Conference on Computer Vision.

[8]  Latifur Khan,et al.  Image annotations by combining multiple evidence & wordNet , 2005, ACM Multimedia.

[9]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[10]  Rafael Muñoz-Salinas,et al.  Example-based procedural modelling by geometric constraint solving , 2011, Multimedia Tools and Applications.

[11]  Alan F. Smeaton,et al.  Semantics-based selection of everyday concepts in visual lifelogging , 2012, International Journal of Multimedia Information Retrieval.

[12]  Paul Over,et al.  High-level feature detection from video in TRECVid: a 5-year retrospective of achievements , 2009 .

[13]  Yi Wu,et al.  Ontology-based multi-classification learning for video concept detection , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[14]  Alan F. Smeaton,et al.  Using visual lifelogs to automatically characterize everyday activities , 2013, Inf. Sci..

[15]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[16]  Alan F. Smeaton,et al.  Factorizing Time-Aware Multi-way Tensors for Enhancing Semantic Wearable Sensing , 2015, MMM.

[17]  Nicu Sebe,et al.  MultiMedia Modeling - 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part I , 2016, MMM.

[18]  Steve Hodges,et al.  SenseCam: A wearable camera that stimulates and rehabilitates autobiographical memory , 2011, Memory.

[19]  Shih-Fu Chang,et al.  Context-Based Concept Fusion with Boosted Conditional Random Fields , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[20]  Chong-Wah Ngo,et al.  Fast Semantic Diffusion for Large-Scale Context-Based Image and Video Annotation , 2012, IEEE Transactions on Image Processing.

[21]  Lifeng Sun,et al.  Towards Training-Free Refinement for Semantic Indexing of Visual Media , 2016, MMM.

[22]  Changhu Wang,et al.  Image annotation refinement using random walk with restarts , 2006, MM '06.