Search Strategies for Pattern Identification in Multimodal Data: Three Case Studies

The analysis of multimodal data benefits from meaningful search and retrieval. This paper investigates strategies of searching multimodal data for event patterns. Through three longitudinal case studies, we observed researchers exploring and identifying event patterns in multimodal data. The events were extracted from different multimedia signal sources ranging from annotated video transcripts to interaction logs. Each researcher's data has varying temporal characteristics (e.g., sparse, dense, or clustered) that posed several challenges for identifying relevant patterns. We identify unique search strategies and better understand the aspects that contributed to each.

[1]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[2]  Ada Wai-Chee Fu,et al.  Discovering Temporal Patterns for Interval-Based Events , 2000, DaWaK.

[3]  Paul R. Cohen,et al.  Fluent Learning: Elucidating the Structure of Episodes , 2001, IDA.

[4]  Mary P. Harper,et al.  VACE Multimodal Meeting Corpus , 2005, MLMI.

[5]  Dmitry Vetrov,et al.  The Algorithm for Detection of Fuzzy Behavioral Patterns , 2010 .

[6]  Roddy Cowie,et al.  AVEC 2012: the continuous audio/visual emotion challenge - an introduction , 2012, ICMI.

[7]  Oliver Brdiczka,et al.  Temporal task footprinting: identifying routine tasks by their temporal patterns , 2010, IUI '10.

[8]  P. Pirolli,et al.  The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis , 2007 .

[9]  Yale Song,et al.  Multimodal human behavior analysis: learning correlation and interaction across modalities , 2012, ICMI '12.

[10]  M S Magnusson,et al.  Discovering hidden time patterns in behavior: T-patterns and their detection , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[11]  Jean Carletta,et al.  The AMI Meeting Corpus: A Pre-announcement , 2005, MLMI.

[12]  P. S. Sastry,et al.  Conditional Probability-Based Significance Tests for Sequential Patterns in Multineuronal Spike Trains , 2008, Neural Computation.

[13]  Debprakash Patnaik,et al.  Inferring neuronal network connectivity from spike data: A temporal data mining approach , 2008, Sci. Program..

[14]  Maja Pantic,et al.  The SEMAINE corpus of emotionally coloured character interactions , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[15]  Francis K. H. Quek,et al.  Structural and temporal inference search (STIS): pattern identification in multimodal data , 2012, ICMI '12.

[16]  Francis K. H. Quek,et al.  Toward multimodal situated analysis , 2011, ICMI '11.

[17]  Ezequiel Morsella,et al.  Expressing oneself/expressing one's self : communication, cognition, language, and identity , 2009 .

[18]  John F. Roddick,et al.  Mining Relationships Between Interacting Episodes , 2004, SDM.

[19]  David McNeill,et al.  Gesture, Gaze, and Ground , 2005, MLMI.

[20]  Francis K. H. Quek,et al.  Interactive relevance search and modeling: support for expert-driven analysis of multimodal data , 2013, ICMI '13.

[21]  Ingo Schmitt,et al.  PythiaSearch: a multiple search strategy-supportive multimedia retrieval system , 2012, ICMR '12.

[22]  Eddie Schwalb,et al.  Temporal Constraints: A Survey , 1998, Constraints.

[23]  Christian Freksa,et al.  Temporal Reasoning Based on Semi-Intervals , 1992, Artif. Intell..

[24]  Francis K. H. Quek,et al.  Structural model discovery in temporal event data streams , 2013 .

[25]  Karrie Karahalios,et al.  VCode and VData: illustrating a new framework for supporting the video annotation workflow , 2008, AVI '08.

[26]  Dmitriy Fradkin,et al.  Robust Mining of Time Intervals with Semi-interval Partial Order Patterns , 2010, SDM.

[27]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[28]  P. S. Sastry,et al.  A survey of temporal data mining , 2006 .

[29]  Michael Kipp,et al.  An Exchange Format for Multimodal Annotations , 2008, LREC.

[30]  Fabian Mörchen,et al.  Unsupervised pattern mining from symbolic temporal data , 2007, SKDD.

[31]  Valerie L. Shalin,et al.  Cognitive task analysis , 2000 .

[32]  Francis K. H. Quek,et al.  Interactive data-driven discovery of temporal behavior models from events in media streams , 2012, ACM Multimedia.

[33]  Teruko Mitamura,et al.  Multimodal knowledge-based analysis in multimedia event detection , 2012, ICMR '12.

[34]  Fabian Mörchen,et al.  Algorithms for time series knowledge mining , 2006, KDD '06.