Wize Mirror - a smart, multisensory cardio-metabolic risk monitoring system

A multi-sensor device for health self-monitoring and assessment is proposed.A real-time head pose estimation and tracking method is introduced.An inexpensive 3D scanner facilitating facial morphology analysis is described.Face 3D shape analysis facilitates tracking changes in weight and BMI index.The evaluation of stress and anxiety seems possible using dynamic facial features. In the recent years personal health monitoring systems have been gaining popularity, both as a result of the pull from the general population, keen to improve well-being and early detection of possibly serious health conditions and the push from the industry eager to translate the current significant progress in computer vision and machine learning into commercial products. One of such systems is the Wize Mirror, built as a result of the FP7 funded SEMEOTICONS (SEMEiotic Oriented Technology for Individuals CardiOmetabolic risk self-assessmeNt and Self-monitoring) project. The project aims to translate the semeiotic code of the human face into computational descriptors and measures, automatically extracted from videos, multispectral images, and 3D scans of the face. The multisensory platform, being developed as the result of that project, in the form of a smart mirror, looks for signs related to cardio-metabolic risks. The goal is to enable users to self-monitor their well-being status over time and improve their life-style via tailored user guidance. This paper is focused on the description of the part of that system, utilising computer vision and machine learning techniques to perform 3D morphological analysis of the face and recognition of psycho-somatic status both linked with cardio-metabolic risks. The paper describes the concepts, methods and the developed implementations as well as reports on the results obtained on both real and synthetic datasets.

[1]  Frédo Durand,et al.  Detecting Pulse from Head Motions in Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Jean-Luc Dugelay,et al.  Building the space scale or how to weigh a person with no gravity , 2012, 2012 IEEE International Conference on Emerging Signal Processing Applications.

[3]  Jongmoo Choi,et al.  Real-time 3-D face tracking and modeling framework for mid-res cam , 2014, IEEE Winter Conference on Applications of Computer Vision.

[4]  Nozomu Hamada,et al.  Mouth motion analysis with space-time interest points , 2009, TENCON 2009 - 2009 IEEE Region 10 Conference.

[5]  Jean-Luc Dugelay,et al.  Weight estimation from visual body appearance , 2010, 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[6]  Roland Wismüller,et al.  European Projects , 2000, Euro-Par.

[7]  Michael Werman,et al.  Robust head pose estimation by fusing time-of-flight depth and color , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[8]  J. Y. Kim,et al.  Predicting visceral obesity based on facial characteristics , 2014, BMC Complementary and Alternative Medicine.

[9]  Bogdan J. Matuszewski,et al.  Head pose tracking for immersive applications , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[10]  Tamás D. Gedeon,et al.  Objective measures, sensors and computational techniques for stress recognition and classification: A survey , 2012, Comput. Methods Programs Biomed..

[11]  Yan Han,et al.  An Interval-Parameter Fuzzy Linear Programming with Stochastic Vertices Model for Water Resources Management under Uncertainty , 2013 .

[12]  Eirini Christinaki,et al.  Facial Signs and Psycho-physical Status Estimation for Well-being Assessment , 2014, HEALTHINF.

[13]  Eirini Christinaki,et al.  Extraction of facial features as indicators of stress and anxiety , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[14]  Eirini Christinaki,et al.  Contactless detection of facial signs related to stress: A preliminary study , 2014, 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH).

[15]  Eirini Christinaki,et al.  Comparison of blind source separation algorithms for optical heart rate monitoring , 2014, 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH).

[16]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[18]  M. Hamilton.  The assessment of anxiety states by rating. , 1959, The British journal of medical psychology.

[19]  Debbie A Lawlor,et al.  A population-based cross-sectional study of the association between facial morphology and cardiometabolic risk factors in adolescence , 2013, BMJ Open.

[20]  Richard A. Reyment.  An idiosyncratic history of early morphometrics , 1996 .

[21]  Carla Cordivari,et al.  Myoclonic disorders: a practical approach for diagnosis and treatment , 2011, Therapeutic advances in neurological disorders.

[22]  Silvia Biasotti,et al.  Mathematical Tools for Shape Analysis and Description , 2014, Synthesis Lectures on Computer Graphics and Animation.

[23]  Antonio L. Apolinario,et al.  KinectFusion for Faces: Real-Time 3D Face Tracking and Modeling Using a Kinect Camera for a Markerless AR System , 2013 .

[24]  Katsuhiko Sakaue,et al.  Head pose estimation by nonlinear manifold learning , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[25]  T J Cole,et al.  Body shape in American and British adults: between-country and inter-ethnic comparisons , 2008, International Journal of Obesity.

[26]  Peter Hammond,et al.  The use of 3D face shape modelling in dysmorphology , 2007, Archives of Disease in Childhood.

[27]  Alex Pentland,et al.  LAFTER: a real-time face and lips tracker with facial expression recognition , 2000, Pattern Recognit..

[28]  L. Romero,et al.  Physiological stress in ecology: lessons from biomedical research. , 2004, Trends in ecology & evolution.

[29]  Bogdan J. Matuszewski,et al.  Improved 3-D facial representation through statistical shape model , 2010, 2010 IEEE International Conference on Image Processing.

[30]  Jack Wang,et al.  Validation of a 3-dimensional photonic scanner for the measurement of body volumes, dimensions, and percentage body fat. , 2006, The American journal of clinical nutrition.

[31]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[32]  Michalis Vazirgiannis,et al.  c ○ 2001 Kluwer Academic Publishers. Manufactured in The Netherlands. On Clustering Validation Techniques , 2022 .

[33]  Paul Suetens,et al.  meshSIFT: Local surface features for 3D face recognition under expression variations and partial data , 2013, Comput. Vis. Image Underst..

[34]  Tao Tang,et al.  Marching Cubes Algorithm for Fast 3D Modeling of Human Face by Incremental Data Fusion , 2013 .

[35]  V. Ferrario,et al.  Soft tissue facial morphology in obese adolescents: a three-dimensional noninvasive assessment. , 2009, The Angle orthodontist.

[36]  J. Harrigan,et al.  How do you look when feeling anxious? Facial displays of anxiety , 1996 .

[37]  Luc Van Gool,et al.  Real Time Head Pose Estimation from Consumer Depth Cameras , 2011, DAGM-Symposium.

[38]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[39]  I. Liberzon,et al.  The Neurocircuitry of Fear, Stress, and Anxiety Disorders , 2011, Neuropsychopharmacology.

[40]  C. Cacou.  Anthropometry of the head and face , 1995 .

[41]  D'arcy W. Thompson,et al.  On Growth and Form , 1917, Nature.

[42]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Andrea Giachetti,et al.  Robust Automatic Measurement of 3D Scanned Models for the Human Body Fat Estimation , 2015, IEEE Journal of Biomedical and Health Informatics.

[44]  Chaoyang Li,et al.  Increasing Trends in Waist Circumference and Abdominal Obesity among U.S. Adults , 2007, Obesity.

[45]  Günther Greiner,et al.  Automatic reconstruction of personalized avatars from 3D face scans , 2011, Comput. Animat. Virtual Worlds.

[46]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[47]  Han Wang,et al.  A fast and robust head pose estimation system based on depth data , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[48]  Bruce D Johnson,et al.  Facial fat and its relationship to abdominal fat: a marker for insulin resistance? , 2004, Medical hypotheses.

[49]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[50]  Zhengyou Zhang,et al.  3D Deformable Face Tracking with a Commodity Depth Camera , 2010, ECCV.

[51]  Antonis A. Argyros,et al.  Head pose estimation on depth data based on Particle Swarm Optimization , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[52]  Antonio E Nardi,et al.  The role of anxiety in metabolic syndrome , 2012, Expert review of endocrinology & metabolism.

[53]  E. Bromfield,et al.  An Introduction to Epilepsy , 2006 .

[54]  Michael G. Strintzis,et al.  Robust real-time 3D head pose estimation from range data , 2005, Pattern Recognit..

[55]  W. Chiou,et al.  Application of three-dimensional body scanner: observation of prevalence of metabolic syndrome. , 2004, Clinical nutrition.

[56]  L. Farkas.  Anthropometry of the head and face , 1994 .

[57]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[58]  Alex Pentland,et al.  Automatic lipreading by optical-flow analysis , 1989 .

[59]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[60]  Thomas B. Moeslund,et al.  Improved pulse detection from head motions using DCT , 2014, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[61]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[62]  Jun-Hyeong Do,et al.  A Classification Method of Normal and Overweight Females Based on Facial Features for Automated Medical Applications , 2012, Journal of biomedicine & biotechnology.

[63]  Somnath Sengupta,et al.  Lip Localization and Viseme Recognition from Video Sequences , 2007 .

[64]  Jongmoo Choi,et al.  Near laser-scan quality 3-D face reconstruction from a low-quality depth stream , 2015, Image Vis. Comput..

[65]  R. Murison,et al.  Neuroscience and Biobehavioral Reviews Stress Revisited: a Critical Evaluation of the Stress Concept , 2022 .

[66]  Murray B Stein,et al.  Anxiety and depressive symptoms and medical illness among adults with anxiety disorders. , 2015, Journal of psychosomatic research.

[67]  Rainer Stiefelhagen,et al.  Head pose estimation using stereo vision for human-robot interaction , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..