Facial expression recognition using two-tier classification and its application to smart home automation system

With the convergence of smart technologies and advancement in electronic equipment the concept of smart home system swiftly escalates. The idea is to automate the home appliances according to the user requirements without human intervention. After a long tiring day and heavy workloads user will not be in a state of taking out its mobile phone and pressing the buttons for controlling home appliances. Several methods have been proposed in the design of such systems using sensors, biometrics and face detection. This paper proposes a method for detecting human emotions by taking into account the complete facial analysis, suggesting that the emotions can accurately be determined by analyzing eyes, nose and lips separately hence covering a wide range of emotions. The classification is carried out by acquiring the image of user followed by the face detection and segmenting the region of interests (ROI) i.e. eyes, nose and lips for further analysis of emotions. Principle Component Analysis (PCA) along with feature extraction techniques and Support Vector Machines (SVMs) are used for classification of emotion for the said automation system. Policies have been implemented in Java to simulate the home automation environment for testing and validation. At the instant this system has been tested on a single user with 4 basic emotions i.e. sad, anger, happiness and neutral, but this study can be a basis to develop an automated system with variety of emotions for multiple users.

[1]  Jhing-Fa Wang,et al.  Robust Environmental Sound Recognition for Home Automation , 2008, IEEE Transactions on Automation Science and Engineering.

[2]  Huiru Zheng,et al.  Human Activity Detection in Smart Home Environment with Self-Adaptive Neural Networks , 2008, 2008 IEEE International Conference on Networking, Sensing and Control.

[3]  Peter W. McOwan,et al.  A real-time automated system for the recognition of human facial expressions , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  Marcel Worring,et al.  Face detection by aggregated Bayesian network classifiers , 2001, Pattern Recognit. Lett..

[5]  Muttukrishnan Rajarajan,et al.  Privacy-Preserving Multi-Class Support Vector Machine for Outsourcing the Data Classification in Cloud , 2014, IEEE Transactions on Dependable and Secure Computing.

[6]  K. J. Parker,et al.  Psychological stress in childhood and susceptibility to the chronic diseases of aging: moving toward a model of behavioral and biological mechanisms. , 2011, Psychological bulletin.

[7]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[8]  Li Dan,et al.  Cognitive emotion model for eldercare robot in smart home , 2015, China Communications.

[9]  Oksam Chae,et al.  Local Directional Number Pattern for Face Analysis: Face and Expression Recognition , 2013, IEEE Transactions on Image Processing.

[10]  Kaamran Raahemifar,et al.  Optimization of home automation systems based on human motion and behaviour , 2014, 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE).

[11]  Tinghuai Ma,et al.  Context-aware implementation based on CBR for smart home , 2005, WiMob'2005), IEEE International Conference on Wireless And Mobile Computing, Networking And Communications, 2005..

[12]  Oliver Brdiczka,et al.  Detecting Human Behavior Models From Multimodal Observation in a Smart Home , 2009, IEEE Transactions on Automation Science and Engineering.

[13]  Fei Zuo,et al.  Real-time embedded face recognition for smart home , 2001 .

[14]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[16]  Emil M. Petriu,et al.  Dynamic sign language and voice recognition for smart home interactive application , 2013, 2013 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[17]  Zhiliang Wang,et al.  The management system with emotional virtual human based on smart home , 2012, 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery.

[18]  Jiliu Zhou,et al.  Geometric feature based facial expression recognition using multiclass support vector machines , 2009, 2009 IEEE International Conference on Granular Computing.

[19]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Emil M. Petriu,et al.  Dynamic Sign Language Recognition for Smart Home Interactive Application Using Stochastic Linear Formal Grammar , 2015, IEEE Transactions on Instrumentation and Measurement.

[21]  Rong Du,et al.  A context-aware entrance guard in smart home: An event-driven application based on the human motion and face recognition , 2011, The 5th International Conference on Automation, Robotics and Applications.

[22]  Ville Ojansivu,et al.  Blur Insensitive Texture Classification Using Local Phase Quantization , 2008, ICISP.

[23]  Qingshan Liu,et al.  Learning active facial patches for expression analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Ali Ziya Alkar,et al.  IP based home automation system , 2010, IEEE Transactions on Consumer Electronics.

[26]  Pawel Forczmanski,et al.  Simple and Robust Facial Portraits Recognition under Variable Lighting Conditions Based on Two-Dimensional Orthogonal Transformations , 2013, ICIAP.

[27]  Barry Brumitt,et al.  EasyLiving: Technologies for Intelligent Environments , 2000, HUC.

[28]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[29]  Ali Ziya Alkar,et al.  A GSM, internet and speech controlled wireless interactive home automation system , 2006, IEEE Transactions on Consumer Electronics.

[30]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[31]  Mandeep Kaur,et al.  Recognition of Facial Expressions with Principal Component Analysis and Singular Value Decomposition , 2010 .

[32]  Michael Nischt,et al.  Localization based object recognition for smart home environments , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[33]  Suk Lee,et al.  Development of PIR sensor based indoor location detection system for smart home , 2006, 2006 SICE-ICASE International Joint Conference.

[34]  Pradeep K. Atrey,et al.  Smart mirror for ambient home environment , 2007 .

[35]  Mohamed Berkane,et al.  Multimodal biometric systems , 2014, 2014 International Conference on Multimedia Computing and Systems (ICMCS).

[36]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[38]  Chengjun Liu,et al.  A Bayesian Discriminating Features Method for Face Detection , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[40]  Zarul Fitri Zaaba,et al.  Eye 2H: A proposed automated smart home control system for detecting human emotions through facial detection , 2014, The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M).

[41]  Stan Z. Li,et al.  FloatBoost learning and statistical face detection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Shuang-Hua Yang,et al.  A zigbee-based home automation system , 2009, IEEE Transactions on Consumer Electronics.

[43]  Rabab Kreidieh Ward,et al.  Adaptive Region-Based Image Enhancement Method for Robust Face Recognition Under Variable Illumination Conditions , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[44]  James W. Davis,et al.  The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment , 1999, Presence.

[45]  Shiqing Zhang,et al.  Facial expression recognition using local binary patterns and discriminant kernel locally linear embedding , 2012, EURASIP Journal on Advances in Signal Processing.