Video-to-text information fusion evaluation for level 5 user refinement

Video-to-Text (V2T) fusion is an example of coordinating low-level information fusion (LLIF) with high-level information fusion (HLIF) through semantic descriptions of physical information. Using hard (e.g., video) and soft (i.e., text) data fusion affords Level 5 User Refinement of object characterization, target tracking, and situation assessment. Building on our previous video-to-text (V2T) Fusion2014 paper, we extend the method for evaluation of eight tracking methods compared for extraction of semantic information including target number, category, attribute, and direction. Using the CMUSphinx speech-to-text system for semantic parsing of user call-outs, preliminary results show the integration of video tracking and text analysis is better with the compressive tracker (CT) and the Tracking-Learning-Detection (TLD) method. The feature analysis of the CT and TLD demonstrate the ability to associate user call-out text-based semantic descriptors with video exploitation. The results are presented in a visualization tool for rapid production to aid user refinement (HLIF) and object assessment (LLIF) functions.

[1]  Genshe Chen,et al.  Context aided video-to-text information fusion , 2014, 17th International Conference on Information Fusion (FUSION).

[2]  E. Blasch,et al.  Sensor Management Fusion Using Operating Conditions , 2008, 2008 IEEE National Aerospace and Electronics Conference.

[3]  Pierre Valin,et al.  Information fusion measures of effectiveness (MOE) for decision support , 2011, Defense + Commercial Sensing.

[4]  James Llinas,et al.  High Level Information Fusion (HLIF): Survey of models, issues, and grand challenges , 2012, IEEE Aerospace and Electronic Systems Magazine.

[5]  Paul Lamere,et al.  Design of the CMU Sphinx-4 Decoder , 2022 .

[6]  Mubarak Shah,et al.  Learning object motion patterns for anomaly detection and improved object detection , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Larry S. Davis,et al.  AVSS 2011 demo session: A large-scale benchmark dataset for event recognition in surveillance video , 2011, AVSS.

[8]  Erik Blasch,et al.  Overview of Dempster-Shafer and belief function tracking methods , 2013, Defense, Security, and Sensing.

[9]  Ehud Rivlin,et al.  Robust Fragments-based Tracking using the Integral Histogram , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[10]  Zhonghai Wang,et al.  Video-based activity analysis using the L1 tracker on VIRAT data , 2013, 2013 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[11]  Erik Blasch,et al.  Overview of contextual tracking approaches in information fusion , 2013, Defense, Security, and Sensing.

[12]  Genshe Chen,et al.  Vehicle detection in wide area aerial surveillance using Temporal Context , 2013, Proceedings of the 16th International Conference on Information Fusion.

[13]  Erik Blasch,et al.  A Holistic Cloud-Enabled Robotics System for Real-Time Video Tracking Application , 2014 .

[14]  Moshe Kam,et al.  Evidence combination for hard and soft sensor data fusion , 2011, 14th International Conference on Information Fusion.

[15]  Narendra Ahuja,et al.  Robust visual tracking via multi-task sparse learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  William M. Pottenger,et al.  A semi-supervised active learning algorithm for information extraction from textual data: Research Articles , 2005 .

[17]  Tao Wang,et al.  Audio-Visual Feature Fusion for Vehicles Classification in a Surveillance System , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[18]  Ming-Hsuan Yang,et al.  Visual tracking with online Multiple Instance Learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[20]  Yi Liu,et al.  SemiBoost: Boosting for Semi-Supervised Learning , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Gian Luca Foresti,et al.  Selecting classifiers by F-score for real-time video tracking , 2010, 2010 13th International Conference on Information Fusion.

[22]  Eloi Bosse,et al.  High-Level Information Fusion Management and System Design , 2012 .

[23]  Li Bai,et al.  Efficient Minimum Error Bounded Particle Resampling L1 Tracker With Occlusion Detection , 2013, IEEE Transactions on Image Processing.

[24]  Horst Bischof,et al.  Semi-supervised On-Line Boosting for Robust Tracking , 2008, ECCV.

[25]  Erik Blasch,et al.  Automatic Association of Chats and Video Tracks for Activity Learning and Recognition in Aerial Video Surveillance , 2014, Sensors.

[26]  Erik Blasch,et al.  Adaptive context assessment and context management , 2014, 17th International Conference on Information Fusion (FUSION).

[27]  Erik Blasch,et al.  Randomized unscented transform in state estimation of non-Gaussian systems: Algorithms and performance , 2012, 2012 15th International Conference on Information Fusion.

[28]  Gee Wah Ng,et al.  High-level Information Fusion: An Overview , 2013, J. Adv. Inf. Fusion.

[29]  Erik Blasch,et al.  Sensor, User, Mission (SUM) Resource Management and Their Interaction with Level 2/3 Fusion , 2006, 2006 9th International Conference on Information Fusion.

[30]  Richard M. Stern,et al.  The 1997 CMU Sphinx-3 English Broadcast News Transcription System , 1997 .

[31]  Anthony Hoogs,et al.  Video content annotation using visual analysis and a large semantic knowledgebase , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[32]  William M. Pottenger,et al.  Higher Order Naïve Bayes: A Novel Non-IID Approach to Text Classification , 2011, IEEE Transactions on Knowledge and Data Engineering.

[33]  Xiaoqin Zhang,et al.  Block covariance based l1 tracker with a subtle template dictionary , 2013, Pattern Recognit..

[34]  Alun D. Preece,et al.  Integrating hard and soft information sources for D2D using controlled natural language , 2012, 2012 15th International Conference on Information Fusion.

[35]  Erik Blasch,et al.  Issues and Challenges in Situation Assessment (Level 2 Fusion) , 2006, J. Adv. Inf. Fusion.

[36]  Erik Blasch,et al.  Towards unbiased evaluation of uncertainty reasoning: The URREF ontology , 2012, 2012 15th International Conference on Information Fusion.

[37]  Richard T. Antony,et al.  First-principle approach to functionally decomposing the JDL fusion model: Emphasis on soft target data , 2008, 2008 11th International Conference on Information Fusion.

[38]  Anthony Hoogs,et al.  Multi-modal fusion for video understanding , 2001, Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery.

[39]  Shuicheng Yan,et al.  Semi-supervised Learning by Sparse Representation , 2009, SDM.

[40]  David L. Hall,et al.  A synthetic dataset for evaluating soft and hard fusion algorithms , 2011, Defense + Commercial Sensing.

[41]  Erik Blasch,et al.  Extraction of Semantic Activities from Twitter Data , 2013, STIDS.

[42]  Shai Avidan,et al.  Ensemble Tracking , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  William M. Pottenger,et al.  Modeling Microtext with Higher Order Learning , 2013, AAAI Spring Symposium: Analyzing Microtext.

[44]  Werner Retschitzegger,et al.  SEM2 suite — Towards a tool suite for supporting knowledge management in situation awareness systems , 2014, Proceedings of the 2014 IEEE 15th International Conference on Information Reuse and Integration (IEEE IRI 2014).

[45]  Anthony Hoogs,et al.  Detecting rare events in video using semantic primitives with HMM , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[46]  Genshe Chen,et al.  Wide-area motion imagery (WAMI) exploitation tools for enhanced situation awareness , 2012, 2012 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[47]  Lyudmila Mihaylova,et al.  Joint target tracking and classification with particle filtering and mixture Kalman filtering using kinematic radar information , 2006, Digit. Signal Process..

[48]  Erik Blasch,et al.  Ontology alignment in geographical hard-soft information fusion systems , 2010, 2010 13th International Conference on Information Fusion.

[49]  Huchuan Lu,et al.  Visual tracking via adaptive structural local sparse appearance model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[51]  Wei Mei,et al.  Simultaneous tracking and classification: a modularized scheme , 2007, IEEE Transactions on Aerospace and Electronic Systems.

[52]  Anton van den Hengel,et al.  Learning Compact Binary Codes for Visual Tracking , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Chun Yang,et al.  Kalman Filtering with Nonlinear State Constraints , 2009 .

[54]  S. Plano,et al.  DFIG level 5 (user refinement) issues supporting situational assessment reasoning , 2005, 2005 7th International Conference on Information Fusion.

[55]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[56]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Erik Blasch,et al.  Revisiting the JDL model for information exploitation , 2013, Proceedings of the 16th International Conference on Information Fusion.

[58]  Erik Blasch,et al.  Minimum Error Bounded Efficient L1 Tracker with Occlusion Detection (PREPRINT) , 2011 .

[59]  Lei Zhang,et al.  Real-Time Compressive Tracking , 2012, ECCV.

[60]  Erik Blasch,et al.  URREF reliability versus credibility in information fusion (STANAG 2511) , 2013, Proceedings of the 16th International Conference on Information Fusion.

[61]  I. Kadar,et al.  Resource management coordination with level 2/3 fusion issues and challenges [Panel Report] , 2008, IEEE Aerospace and Electronic Systems Magazine.

[62]  Genshe Chen,et al.  Scalable sentiment classification for Big Data analysis using Naïve Bayes Classifier , 2013, 2013 IEEE International Conference on Big Data.

[63]  Erik Blasch,et al.  Fusion of Tracks with Road Constraints , 2008, J. Adv. Inf. Fusion.

[64]  Erik Blasch,et al.  Enhanced air operations using JView for an air-ground fused situation awareness udop , 2013, 2013 IEEE/AIAA 32nd Digital Avionics Systems Conference (DASC).

[65]  Daniel J. Garland,et al.  Situation Awareness Analysis and Measurement , 2009 .

[66]  Thomas S. Huang,et al.  CBIR: from low-level features to high-level semantics , 2000, Electronic Imaging.

[67]  Gian Luca Foresti,et al.  Cascaded online boosting , 2010, Journal of Real-Time Image Processing.

[68]  Roy L. Streit,et al.  Tracking, Association, and Classification: A Combined PMHT Approach , 2002, Digit. Signal Process..

[69]  Horst Bischof,et al.  Real-Time Tracking via On-line Boosting , 2006, BMVC.

[70]  Erik Blasch,et al.  Multi-scale decomposition tool for Content Based Image Retrieval , 2013, 2013 IEEE Applied Imagery Pattern Recognition Workshop (AIPR).

[71]  E. Blasch,et al.  Multiresolution EO/IR target tracking and identification , 2005, 2005 7th International Conference on Information Fusion.

[72]  Alan S. Willsky,et al.  Lessons learned in the creation of a data set for hard/soft information fusion , 2009, 2009 12th International Conference on Information Fusion.

[73]  Erik Blasch,et al.  Multi-source Multi-modal Activity Recognition in Aerial Video Surveillance , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[74]  Erik Blasch,et al.  Measures of effectiveness for high-level fusion , 2010, 2010 13th International Conference on Information Fusion.

[75]  Zhonghai Wang,et al.  Pattern of life from WAMI objects tracking based on visual context-aware tracking and infusion network models , 2013, Defense, Security, and Sensing.

[76]  Li Bai,et al.  Robust infrared vehicle tracking across target pose change using L1 regularization , 2010, 2010 13th International Conference on Information Fusion.

[77]  Li Bai,et al.  Multiple Kernel Learning for vehicle detection in wide area motion imagery , 2012, 2012 15th International Conference on Information Fusion.