IBES: a tool for creating instructions based on event segmentation

Receiving informative, well-structured, and well-designed instructions supports performance and memory in assembly tasks. We describe IBES, a tool with which users can quickly and easily create multimedia, step-by-step instructions by segmenting a video of a task into segments. In a validation study we demonstrate that the step-by-step structure of the visual instructions created by the tool corresponds to the natural event boundaries, which are assessed by event segmentation and are known to play an important role in memory processes. In one part of the study, 20 participants created instructions based on videos of two different scenarios by using the proposed tool. In the other part of the study, 10 and 12 participants respectively segmented videos of the same scenarios yielding event boundaries for coarse and fine events. We found that the visual steps chosen by the participants for creating the instruction manual had corresponding events in the event segmentation. The number of instructional steps was a compromise between the number of fine and coarse events. Our interpretation of results is that the tool picks up on natural human event perception processes of segmenting an ongoing activity into events and enables the convenient transfer into meaningful multimedia instructions for assembly tasks. We discuss the practical application of IBES, for example, creating manuals for differing expertise levels, and give suggestions for research on user-oriented instructional design based on this tool.

[1]  Darren Newtson,et al.  The perceptual organization of ongoing behavior , 1976 .

[2]  Darren Newtson,et al.  The objective basis of behavior units. , 1977 .

[3]  Darren Newtson,et al.  Selective perception of events. , 1979 .

[4]  Andrew P. Witkin,et al.  What Is Perceptual Organization For? , 1983, IJCAI.

[5]  William G. Graziano,et al.  Social Cognition as Segmentation of the Stream of Behavior. , 1988 .

[6]  Harm J. G. Zwaga,et al.  Visual Information For Everyday Use : Design And Research Perspectives , 1999 .

[7]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[8]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[9]  Barbara Tversky,et al.  Structuring information interfaces for procedural learning. , 2003, Journal of experimental psychology. Applied.

[10]  Pat Hanrahan,et al.  Designing effective step-by-step assembly instructions , 2003, ACM Trans. Graph..

[11]  Pat Hanrahan,et al.  Identification and validation of cognitive design principles for automated generation of assembly instructions , 2004, AVI.

[12]  S. Schwan,et al.  The cognitive representation of filmic event summaries , 2004 .

[13]  Brian P. Bailey,et al.  If not now, when?: the effects of interruption at different moments within task execution , 2004, CHI.

[14]  Analysis of Assembly Instructions for Children: The Good, the Bad, and the Ugly , 2005 .

[15]  Richard Mayer,et al.  Multimedia Learning , 2001, Visible Learning Guide to Student Achievement.

[16]  Mary Hegarty,et al.  When static media promote active learning: annotated illustrations versus narrated animations in multimedia instruction. , 2005, Journal of experimental psychology. Applied.

[17]  Maneesh Agrawala,et al.  Cognitive Design Principles for Automated Generation of Visualizations , 2020, Applied Spatial Cognition: From Research to Cognitive Technology.

[18]  Tad T. Brunyé,et al.  Learning Procedures: The Role of Working Memory in Multimedia Learning Experiences , 2022 .

[19]  Stephan Schwan,et al.  The Role of Segmentation in Perception and Understanding of Events , 2008 .

[20]  Rama Chellappa,et al.  Computational Vision Approaches for Event Modeling , 2008 .

[21]  Holger Horz,et al.  Multimedia: How to Combine Language and Visuals , 2008 .

[22]  Jeffrey M. Zacks,et al.  Understanding events : from perception to action , 2008 .

[23]  Jeffrey M. Zacks,et al.  Segmentation in the perception and memory of events , 2008, Trends in Cognitive Sciences.

[24]  J. Zacks,et al.  Using movement and intentions to understand human activity , 2009, Cognition.

[25]  Didier Stricker,et al.  Fast Hand Detection Using Posture Invariant Constraints , 2009, KI.

[26]  Sonja Stork,et al.  A multi-dimensional measure for determining the complexity of manual assembly operations , 2009, Prod. Eng..

[27]  T. Gog,et al.  A Theoretical Analysis of How Segmentation of Dynamic Visualizations Optimizes Students' Learning , 2010 .

[28]  T. Gog,et al.  Cognitive Load Theory: Advances in Research on Worked Examples, Animations, and Cognitive Load Measurement , 2010 .

[29]  Anthony G. Cohn,et al.  Exploiting petri-net structure for activity classification and user instruction within an industrial setting , 2011, ICMI '11.

[30]  Gustaf Hendeby,et al.  Using egocentric vision to achieve robust inertial body tracking under magnetic disturbances , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[31]  Elsa Eiriksdottir,et al.  Procedural Instructions, Principles, and Examples , 2011, Hum. Factors.

[32]  Miriam Woestenenk.  The effect on appreciation by tailoring manuals to mental factors ; Will appreciation be enhanced when avance organizers, the level of chunking and document orientation are tailored to cognitive style, computer self-efficacy and need for cognition? , 2011 .

[33]  Tamara van Gog,et al.  Explaining the segmentation effect in learning from animations: The role of pausing and temporal cueing , 2012, Comput. Educ..

[34]  Katharina Scheiter,et al.  Examining learning from text and pictures for different task types: Does the multimedia effect differ for conceptual, causal, and procedural tasks? , 2012, Comput. Hum. Behav..

[35]  Markus Huff,et al.  Visual target detection is impaired at event boundaries , 2012 .

[36]  Didier Stricker,et al.  Learning task structure from video examples for workflow tracking and authoring , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[37]  Barbara Tversky,et al.  How to put things together , 2012, Cognitive Processing.

[38]  Markus Huff,et al.  The verbal facilitation effect in learning to tie nautical knots , 2012 .

[39]  Dominic Gorecky,et al.  Spielebasiertes Training gestalten und integrieren , 2013 .