Interest Level Estimation Based on Tensor Completion via Feature Integration for Partially Paired User’s Behavior and Videos

A novel method for interest level estimation based on tensor completion via feature integration for partially paired users’ behavior and videos is presented in this paper. The proposed method defines a novel canonical correlation analysis (CCA) framework that is suitable for interest level estimation, which is a hybrid version of semi-supervised CCA (SemiCCA) and supervised locality preserving CCA (SLPCCA) called semi-supervised locality preserving CCA (S2LPCCA). For partially paired users’ behavior and videos in actual shops and on the Internet, new integrated features that maximize the correlation between partially paired samples by the principal component analysis (PCA)-mixed CCA framework are calculated. Then videos that users have not watched can be used for the estimation of users’ interest levels. Furthermore, local structures of partially paired samples in the same class are preserved for accurate estimation of interest levels. Tensor completion, which can be applied to three contexts, videos, users and “canonical features and interest levels,” is used for estimation of interest levels. Consequently, the proposed method realizes accurate estimation of users’ interest levels based on S2LPCCA and the tensor completion from partially paired training features of users’ behavior and videos. Experimental results obtained by applying the proposed method to actual data show the effectiveness of the proposed method.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  M. Shamim Hossain,et al.  Emotion-Aware Video QoE Assessment Via Transfer Learning , 2019, IEEE MultiMedia.

[3]  Alvy Ray Smith,et al.  Color gamut transform pairs , 1978, SIGGRAPH.

[4]  Jihoon Yang,et al.  A deep learning based video classification system using multimodality correlation approach , 2017, 2017 17th International Conference on Control, Automation and Systems (ICCAS).

[5]  Yanlei Gu,et al.  Customer behavior classification using surveillance camera for marketing , 2017, Multimedia Tools and Applications.

[6]  Quan-Sen Sun,et al.  A novel semi-supervised canonical correlation analysis and extensions for multi-view dimensionality reduction , 2014, J. Vis. Commun. Image Represent..

[7]  Yanlei Gu,et al.  Customer Behavior Recognition in Retail Store from Surveillance Camera , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[8]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Léon J. M. Rothkrantz,et al.  Semantic assessment of shopping behavior using trajectories, shopping related actions, and context information , 2013, Pattern Recognit. Lett..

[10]  Yingxu Wang,et al.  Kinect Sensor Gesture and Activity Recognition: New Applications for Consumer Cognitive Systems , 2018, IEEE Consumer Electronics Magazine.

[11]  Kyoung-Woon On,et al.  Temporal Attention Mechanism with Conditional Inference for Large-Scale Multi-label Video Classification , 2018, ECCV Workshops.

[12]  Yin Zhang,et al.  Fairness-Aware Recommendation of Information Curators , 2018, ArXiv.

[13]  J. Henderson.  Human gaze control during real-world scene perception , 2003, Trends in Cognitive Sciences.

[14]  Shumin Zhai,et al.  Conversing with the user based on eye-gaze patterns , 2005, CHI.

[15]  Ling Guan,et al.  Joint intermodal and intramodal correlation preservation for semi-paired learning , 2018, Pattern Recognit..

[16]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Jianmin Jiang,et al.  Human Eye Movements Reveal Video Frame Importance , 2019, Computer.

[18]  You-Chiun Wang,et al.  3S-cart: A Lightweight, Interactive Sensor-Based Cart for Smart Shopping in Supermarkets , 2016, IEEE Sensors Journal.

[19]  James Caverlee,et al.  Tensor Completion Algorithms in Big Data Analytics , 2017, ACM Trans. Knowl. Discov. Data.

[20]  Muhammad Zeeshan Khan,et al.  Story Based Video Retrieval using Deep Visual and Textual Information , 2019, 2019 2nd International Conference on Communication, Computing and Digital systems (C-CODE).

[21]  Johan A. K. Suykens,et al.  Regularized Semipaired Kernel CCA for Domain Adaptation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Mohamed Atri,et al.  An efficient end-to-end deep learning architecture for activity classification , 2018, Analog Integrated Circuits and Signal Processing.

[23]  Hyun Myung,et al.  Weighted joint-based human behavior recognition algorithm using only depth information for low-cost intelligent video-surveillance system , 2016, Expert Syst. Appl..

[24]  Raymond R. Burke,et al.  Modeling the effects of dynamic group influence on shopper zone choice, purchase conversion, and spending , 2018, Journal of the Academy of Marketing Science.

[25]  Miki Haseyama,et al.  Interest Level Estimation of Items via Matrix Completion Based on Adaptive User Matrix Construction , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[26]  Deep Medhi,et al.  Measurement of Quality of Experience of Video-on-Demand Services: A Survey , 2016, IEEE Communications Surveys & Tutorials.

[27]  Yanjiao Chen,et al.  From QoS to QoE: A Tutorial on Video Quality Assessment , 2015, IEEE Communications Surveys & Tutorials.

[28]  Anton Nijholt,et al.  Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes , 2001, CHI.

[29]  Yang Feng,et al.  A Central-Scotoma Simulator Based on Low-Cost Eye Tracker , 2018, 2018 IEEE International Conference on Mechatronics and Automation (ICMA).

[30]  Stephen H. Fairclough,et al.  Classification Accuracy from the Perspective of the User: Real-Time Interaction with Physiological Computing , 2015, CHI.

[31]  Jinglei Lv,et al.  What Makes a Good Movie Trailer?: Interpretation from Simultaneous EEG and Eyetracker Recording , 2016, ACM Multimedia.

[32]  Gang Ma,et al.  Semi-paired Probabilistic Canonical Correlation Analysis , 2014, Intelligent Information Processing.

[33]  J. Pettersson,et al.  Cognitive Ability Evaluation using Virtual Reality and Eye Tracking , 2018, 2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA).

[34]  Atul Prakash,et al.  Robust Physical-World Attacks on Deep Learning Visual Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Hiroyuki Kidokoro,et al.  Effectiveness of Cooperative Customer Navigation from Robots around a Retail Shop , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[36]  Mitsuji Muneyasu,et al.  Video Retrieval by Reranking and Relevance Feedback with Tag-Based Similarity , 2018, 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE).

[37]  Ruby B. Lee,et al.  Implicit Sensor-based Authentication of Smartphone Users with Smartwatch , 2016, HASP 2016.

[38]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Andrew Chi-Sing Leung,et al.  Sparse and Truncated Nuclear Norm Based Tensor Completion , 2017, Neural Processing Letters.

[41]  Ling Zou,et al.  Unsupervised Video Highlight Extraction via Query-related Deep Transfer , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[42]  Hong Yan,et al.  Tagrec-CMTF: Coupled Matrix and Tensor Factorization for Tag Recommendation , 2018, IEEE Access.

[43]  Xu Zhang,et al.  Feature-level fusion of fingerprint and finger-vein for personal identification , 2012, Pattern Recognit. Lett..

[44]  Xiaohong Chen,et al.  A unified dimensionality reduction framework for semi-paired and semi-supervised multi-view data , 2012, Pattern Recognit..

[45]  Ming Yang,et al.  Cost Sensitive Semi-Supervised Canonical Correlation Analysis for Multi-view Dimensionality Reduction , 2016, Neural Processing Letters.

[46]  Ale Smidts,et al.  Brain Responses to Movie Trailers Predict Individual Preferences for Movies and Their Population-Wide Commercial Success , 2015 .

[47]  Jiuyang Tang,et al.  Efficient and Accurate Traffic Flow Prediction via Incremental Tensor Completion , 2018, IEEE Access.

[48]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[49]  Xuelong Li,et al.  Matrix completion by Truncated Nuclear Norm Regularization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[51]  Tatsuya Harada,et al.  Generalized Bayesian Canonical Correlation Analysis with Missing Modalities , 2018, ECCV Workshops.

[52]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[53]  Aiguo Song,et al.  Interested Object Detection based on Gaze using Low-cost Remote Eye Tracker , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[54]  Heng Tao Shen,et al.  Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning , 2017, IJCAI.

[55]  Yue Ding,et al.  Inter-Brain EEG Feature Extraction and Analysis for Continuous Implicit Emotion Tagging During Video Watching , 2021, IEEE Transactions on Affective Computing.

[56]  Heng Liu,et al.  Sparse regularized discriminative canonical correlation analysis for multi-view semi-supervised learning , 2018, Neural Computing and Applications.

[57]  Miki Haseyama,et al.  Personalized video preference estimation based on early fusion using multiple users' viewing behavior , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[58]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Hirokazu Kameoka,et al.  SemiCCA: Efficient Semi-supervised Learning of Canonical Correlations , 2010, International Conference on Pattern Recognition.