Probabilistic modelling and inference of human behaviour from mobile phone time series

With an estimated 4.1 billion subscribers around the world, the mobile phone offers a unique opportunity to sense and understand human behaviour from location, co-presence and communication data. While the benefit of modelling this unprecedented amount of data is widely recognised, a number of challenges impede the development of accurate behaviour models. In this thesis, we identify and address two modelling problems and show that their consideration improves the accuracy of behaviour inference. We first examine the modelling of long-range dependencies in human behaviour. Human behaviour models only take into account short-range dependencies in mobile phone time series. Using information theory, we quantify long-range dependencies in mobile phone time series for the first time, demonstrate that they exhibit periodic oscillations and introduce novel tools to analyse them. We further show that considering what the user did 24 hours earlier improves accuracy when predicting user behaviour five hours or longer in advance. The second problem that we address is the modelling of temporal variations in human behaviour. The time spent by a user on an activity varies from one day to the next. In order to recognise behaviour patterns despite temporal variations, we establish a methodological connection between human behaviour modelling and biological sequence alignment. This connection allows us to compare, cluster and model behaviour sequences and introduce novel features for behaviour recognition which improve its accuracy. The experiments presented in this thesis have been conducted on the largest publicly available mobile phone dataset labelled in an unsupervised fashion and are entirely repeatable. Furthermore, our techniques only require cellular data which can easily be recorded by today’s mobile phones and could benefit a wide range of applications including life logging, health monitoring, customer profiling and large-scale surveillance.

[1]  William J. McGill.  Multivariate information transmission , 1954, Trans. IRE Prof. Group Inf. Theory.

[2]  Robert R. Sokal,et al.  A statistical method for evaluating systematic relationships , 1958 .

[3]  S. Katz,et al.  STUDIES OF ILLNESS IN THE AGED. THE INDEX OF ADL: A STANDARDIZED MEASURE OF BIOLOGICAL AND PSYCHOSOCIAL FUNCTION. , 1963, JAMA.

[4]  Andrew J. Viterbi,et al.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.

[5]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[6]  N. S. Sridharan,et al.  The Plan Recognition Problem: An Intersection of Psychology and Artificial Intelligence , 1978, Artif. Intell..

[7]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[8]  Sydney Katz.  Assessing Self‐maintenance: Activities of Daily Living, Mobility, and Instrumental Activities of Daily Living , 1983, Journal of the American Geriatrics Society.

[9]  Fraser,et al.  Independent coordinates for strange attractors from mutual information. , 1986, Physical review. A, General physics.

[10]  Henry A. Kautz.  A formal theory of plan recognition , 1987 .

[11]  J. Rodgers,et al.  Thirteen ways to look at the correlation coefficient , 1988 .

[12]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[13]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[14]  Usama M. Fayyad,et al.  Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning , 1993, IJCAI.

[15]  Geoffrey Leech,et al.  CLAWS4: The Tagging of the British National Corpus , 1994, COLING.

[16]  Oren Etzioni,et al.  A Sound and Fast Goal Recognizer , 1995, IJCAI.

[17]  S. V. Vaseghi.  State duration modelling in hidden Markov models , 1995, Signal Process..

[18]  Pedro M. Domingos,et al.  Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier , 1996, ICML.

[19]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[20]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[21]  Anil K. Jain,et al.  Artificial Neural Networks: A Tutorial , 1996, Computer.

[22]  Zoubin Ghahramani,et al.  Learning Dynamic Bayesian Networks , 1997, Summer School on Neural Networks.

[23]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[24]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Sean R. Eddy,et al.  Profile hidden Markov models , 1998, Bioinform..

[27]  Michael D. Garris,et al.  Neural network-based systems for handprint OCR applications , 1998, IEEE Trans. Image Process..

[28]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[29]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[30]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[31]  Gaetano Borriello,et al.  Location Systems for Ubiquitous Computing , 2001, Computer.

[32]  Maurizio A. Spirito,et al.  On the accuracy of cellular mobile station location estimation , 2001, IEEE Trans. Veh. Technol..

[33]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[34]  Florian Michahelles,et al.  Proactive Instructions for Furniture Assembly , 2002, UbiComp.

[35]  N. Papanikolopoulos,et al.  Vision-Based Human Tracking and Activity Recognition , 2003 .

[36]  Jörg Schultz,et al.  HMM Logos for visualization of protein families , 2004, BMC Bioinformatics.

[37]  T. Speed,et al.  Biological Sequence Analysis , 1998 .

[38]  Lars Erik Holmquist,et al.  Sonic City: The Urban Environment as a Musical Interface , 2003, NIME.

[39]  Harry Zhang,et al.  The Optimality of Naive Bayes , 2004, FLAIRS.

[40]  Stephen J. McKenna,et al.  Activity summarisation and fall detection in a supportive home environment , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[41]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[42]  A. Karimi,et al.  Master‟s thesis , 2011 .

[43]  Wei Niu,et al.  Human activity detection and recognition for video surveillance , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[44]  Walter Bender,et al.  Next-Generation Personal Memory Aids , 2004 .

[45]  Gerhard Rigoll,et al.  Action segmentation and recognition in meeting room scenarios , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[46]  Eric Horvitz,et al.  S-SEER: Selective Perception in a Multimodal Office Activity Recognition System , 2004, MLMI.

[47]  Lars Erik Holmquist,et al.  In Duet with Everyday Urban Settings: A User Study of Sonic City , 2004, NIME.

[48]  Henry A. Kautz,et al.  Inferring activities from interactions with objects , 2004, IEEE Pervasive Computing.

[49]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[50]  Krzysztof Z. Gajos,et al.  Opportunity Knocks: A System to Provide Cognitive Assistance with Transportation Services , 2004, UbiComp.

[51]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[52]  D. Kibler,et al.  Instance-based learning algorithms , 2004, Machine Learning.

[53]  Matthai Philipose,et al.  Mining models of human activities from the web , 2004, WWW '04.

[54]  M. Harville,et al.  Fast, integrated person tracking and activity recognition with plan-view templates from a single stereo camera , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[55]  Michel Joseph Lambert.  Visualizing and analyzing human-centered data streams , 2005 .

[56]  Mika Raento,et al.  ContextPhone: a prototyping platform for context-aware mobile applications , 2005, IEEE Pervasive Computing.

[57]  Simon King,et al.  MMM2: mobile media metadata for photo sharing , 2005, MULTIMEDIA '05.

[58]  Samy Bengio,et al.  Automatic analysis of multimodal group actions in meetings , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Bill N. Schilit,et al.  Place Lab: Device Positioning Using Radio Beacons in the Wild , 2005, Pervasive.

[60]  Alex Pentland,et al.  Reality mining: sensing complex social systems , 2006, Personal and Ubiquitous Computing.

[61]  Simon King,et al.  Towards context-aware face recognition , 2005, MULTIMEDIA '05.

[62]  Henry A. Kautz,et al.  Fine-grained activity recognition by aggregating abstract object usage , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[63]  Shouyang Wang,et al.  Forecasting stock market movement direction with support vector machine , 2005, Comput. Oper. Res..

[64]  Bir Bhanu,et al.  Human Activity Recognition in Thermal Infrared Imagery , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[65]  I. Anderson,et al.  Context Awareness via GSM Signal Strength Fluctuation ? , 2006 .

[66]  Bill Tomlinson,et al.  PersonalSoundtrack: context-aware playlists that adapt to user pace , 2006, CHI Extended Abstracts.

[67]  Fred Stentiford,et al.  Using context and similarity for face and location identification , 2006, Electronic Imaging.

[68]  Sung-Bae Cho,et al.  A Context-Aware Music Recommendation System Using Fuzzy Bayesian Networks with Utility Theory , 2006, FSKD.

[69]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[71]  Anthony Rowe,et al.  Location and Activity Recognition Using eWatch: A Wearable Sensor Platform , 2006, Ambient Intelligence in Everyday.

[72]  Tristan Henderson,et al.  CRAWDAD: a community resource for archiving wireless data at Dartmouth , 2005, CCRV.

[73]  William G. Griswold,et al.  Mobility Detection Using Everyday GSM Traces , 2006, UbiComp.

[74]  Gaetano Borriello,et al.  A Practical Approach to Recognizing Physical Activities , 2006, Pervasive.

[75]  Ian D. Reid,et al.  A general method for human activity recognition in video , 2006, Comput. Vis. Image Underst..

[76]  Dieter Fox,et al.  Location-Based Activity Recognition , 2005, KI.

[77]  Allen R. Hanson,et al.  Aging in place: fall detection and localization in a distributed smart camera network , 2007, ACM Multimedia.

[78]  Haifeng Li,et al.  Adaptive Learning of Semantic Locations and Routes , 2007, LoCA.

[79]  Tim Oates,et al.  A Human Activity Aware Learning Mobile Music Player , 2007, AITamI@IJCAI.

[80]  Henry A. Kautz,et al.  Learning and inferring transportation routines , 2004, Artif. Intell..

[81]  Ahmad Rahmati,et al.  Context-for-wireless: context-sensitive energy-efficient wireless data transfer , 2007, MobiSys '07.

[82]  Emiliano Miluzzo,et al.  CenceMe - Injecting Sensing Presence into Social Networking Applications , 2007, EuroSSC.

[83]  James M. Rehg,et al.  A Scalable Approach to Activity Recognition based on Object Use , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[84]  Sung-Bae Cho,et al.  AniDiary: Daily Cartoon-Style Diary Exploits Bayesian Networks , 2007, IEEE Pervasive Computing.

[85]  Katayoun Farrahi,et al.  Daily Routine Classification from Mobile Phone Data , 2008, MLMI.

[86]  James A. Landay,et al.  The Mobile Sensing Platform: An Embedded Activity Recognition System , 2008, IEEE Pervasive Computing.

[87]  Venet Osmani,et al.  Human activity recognition in pervasive health-care: Supporting efficient remote collaboration , 2008, J. Netw. Comput. Appl..

[88]  Henry Kautz,et al.  Hierarchical recognition of activities of daily living using multi-scale, multi-perspective vision and RFID , 2008 .

[89]  Daniel Gatica-Perez,et al.  What did you do today?: discovering daily routines from large-scale mobile data , 2008, ACM Multimedia.

[90]  Paul Lukowicz,et al.  Wearable Activity Tracking in Car Manufacturing , 2008, IEEE Pervasive Computing.

[91]  Mirco Musolesi,et al.  Sensing meets mobile social networks: the design, implementation and evaluation of the CenceMe application , 2008, SenSys '08.

[92]  Yolande Berbers,et al.  Mobile phones assisting with health self-care: a diabetes case study , 2008, Mobile HCI.

[93]  Kent Larson,et al.  Adding GPS-Control to Traditional Thermostats: An Exploration of Potential Energy Savings and Design Challenges , 2009, Pervasive.

[94]  A. Pentland,et al.  Eigenbehaviors: identifying structure in routine , 2009, Behavioral Ecology and Sociobiology.

[95]  John A. Quinn,et al.  Methodologies for Continuous Cellular Tower Data Analysis , 2009, Pervasive.

[96]  Naranker Dulay,et al.  Activity Inference through Sequence Alignment , 2009, LoCA.

[97]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.

[98]  I. Anderson,et al.  Practical Activity Recognition using GSM Data ∗ , .

[99]  Christus,et al.  A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins , 2022 .