LPaMI: A Graph-Based Lifestyle Pattern Mining Application Using Personal Image Collections in Smartphones

Normally, individuals use smartphones for a variety of purposes like photography, schedule planning, playing games, and so on, apart from benefiting from the core tasks of call-making and short messaging. These services are sources of personal data generation. Therefore, any application that utilises personal data of a user from his/her smartphone is truly a great witness of his/her interests and this information can be used for various personalised services. In this paper, we present Lifestyle Pattern MIning (LPaMI), which is a personalised application for mining the lifestyle patterns of a smartphone user. LPaMI uses the personal photograph collections of a user, which reflect the day-to-day photos taken by a smartphone, to recognise scenes (called objects of interest in our work). These are then mined to discover lifestyle patterns. The uniqueness of LPaMI lies in our graph-based approach to mining the patterns of interest. Modelling of data in the form of graphs is effective in preserving the lifestyle behaviour maintained over the passage of time. Graph-modelled lifestyle data enables us to apply variety of graph mining techniques for pattern discovery. To demonstrate the effectiveness of our proposal, we have developed a prototype system for LPaMI to implement its end-to-end pipeline. We have also conducted an extensive evaluation for various phases of LPaMI using different real-world datasets. We understand that the output of LPaMI can be utilised for variety of pattern discovery application areas like trip and food recommendations, shopping, and so on.

[1]  Changxin Gao,et al.  Spatial multi-scale gradient orientation consistency for place instance and Scene category recognition , 2016, Inf. Sci..

[2]  Takumi Kobayashi,et al.  Image Feature Extraction Using Gradient Local Auto-Correlations , 2008, ECCV.

[3]  Saeed Moghaddam,et al.  MobileMiner: mining your frequent patterns on your phone , 2014, UbiComp.

[4]  Fanglin Chen,et al.  StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones , 2014, UbiComp.

[5]  James M. Rehg,et al.  Where am I: Place instance and category recognition using spatial PACT , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Evaggelos Spyrou,et al.  Xenia: A context aware tour recommendation system based on social network metadata information , 2016, 2016 11th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP).

[7]  Adriana Giret,et al.  A Multi Agent Architecture for Tourism Recommendation , 2010, PAAMS.

[8]  Fangyuan Wang,et al.  Global and Local Features based topic model for scene recognition , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[9]  Thar Baker,et al.  Micro-context recognition of sedentary behaviour using smartphone , 2016, 2016 Sixth International Conference on Digital Information and Communication Technology and its Applications (DICTAP).

[10]  Jiebo Luo,et al.  Towards Lifestyle Understanding: Predicting Home and Vacation Locations from User's Online Photo Collections , 2015, ICWSM.

[11]  Diane J. Cook,et al.  Mining and monitoring patterns of daily routines for assisted living in real world settings , 2010, IHI.

[12]  Daniel Gatica-Perez,et al.  Smartphone usage in the wild: a large-scale analysis of applications and context , 2011, ICMI '11.

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Florent Perronnin,et al.  Large-scale image retrieval with compressed Fisher vectors , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Jiong Yang,et al.  SPIN: mining maximal frequent subgraphs from graph databases , 2004, KDD.

[16]  Frans Coenen,et al.  A survey of frequent subgraph mining algorithms , 2012, The Knowledge Engineering Review.

[17]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[18]  Jiawei Han,et al.  gSpan: graph-based substructure pattern mining , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[19]  Jing Wang,et al.  LIPS: Lifestyle Learning via Mobile Phone Sensing , 2016, 2016 IEEE Global Communications Conference (GLOBECOM).

[20]  George Karypis,et al.  Finding Frequent Patterns in a Large Sparse Graph* , 2005, Data Mining and Knowledge Discovery.

[21]  Panos Kalnis,et al.  GRAMI: Frequent Subgraph and Pattern Mining in a Single Large Graph , 2014, Proc. VLDB Endow..

[22]  George Karypis,et al.  GREW - a scalable frequent subgraph discovery algorithm , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[23]  Takashi Washio,et al.  An Apriori-Based Algorithm for Mining Frequent Substructures from Graph Data , 2000, PKDD.

[24]  Satchidananda Panda,et al.  A camera-phone based study reveals erratic eating pattern and disrupted daily eating-fasting cycle among adults in India , 2017, PloS one.

[25]  Svetlana Lazebnik,et al.  Multi-scale Orderless Pooling of Deep Convolutional Activation Features , 2014, ECCV.

[26]  Joost N. Kok,et al.  The Gaston Tool for Frequent Subgraph Mining , 2005, GraBaTs.

[27]  Luigi Ceccaroni,et al.  PaTac: Urban, Ubiquitous, Personalized Services for Citizens and Tourists , 2009, 2009 Third International Conference on Digital Society.

[28]  Robert Li Kam Wa.  MoodScope: Building a Mood Sensor from Smartphone Usage Patterns , 2012 .

[29]  Steve E Hodges,et al.  Wearable cameras in health: the state of the art and future possibilities. , 2013, American journal of preventive medicine.

[30]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Antonio Torralba,et al.  Contextual Priming for Object Detection , 2003, International Journal of Computer Vision.

[32]  Soundararajan Srinivasan,et al.  Multisensor Fusion in Smartphones for Lifestyle Monitoring , 2010, 2010 International Conference on Body Sensor Networks.

[33]  Elgar Fleisch,et al.  Understanding the impact of personality traits on mobile app adoption - Insights from a large-scale field study , 2016, Comput. Hum. Behav..

[34]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Kai Zheng,et al.  Category co-occurrence modeling for large scale scene recognition , 2016, Pattern Recognit..

[36]  Evaggelos Spyrou,et al.  A survey on Flickr multimedia research challenges , 2016, Eng. Appl. Artif. Intell..

[37]  Muhammad Fahim,et al.  Tracking the sedentary lifestyle using smartphone: A pilot study , 2016, 2016 18th International Conference on Advanced Communication Technology (ICACT).

[38]  Christopher Leckie,et al.  PersTour: A Personalized Tour Recommendation and Planning System , 2016, HT.

[39]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[40]  Andrew T. Campbell,et al.  Bewell: A smartphone application to monitor, model and promote wellbeing , 2011, PervasiveHealth 2011.

[41]  Ying Wah Teh,et al.  Mining Personal Data Using Smartphones and Wearable Devices: A Survey , 2015, Sensors.

[42]  JeongKyu Lee.  A Graph-Based Approach for Modeling and Indexing Video Data , 2006, Eighth IEEE International Symposium on Multimedia (ISM'06).

[43]  Young-Koo Lee,et al.  Top-k frequent induced subgraph mining using sampling , 2016, EDB.

[44]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[45]  Chang-Shing Lee,et al.  Ontological recommendation multi-agent for Tainan City travel , 2009, Expert Syst. Appl..

[46]  J González-Boticario.  SAMAP : An user-oriented adaptive system for planning tourist visits , 2017 .

[47]  Geoff Holmes,et al.  Mining frequent closed graphs on evolving data streams , 2011, KDD.

[48]  Alexander Ilic,et al.  Reality-Mining with Smartphones: Detecting and Predicting Life Events based on App Installation Behavior , 2015, ICIS.

[49]  Robert C. Edgar,et al.  MUSCLE: multiple sequence alignment with high accuracy and high throughput. , 2004, Nucleic acids research.