Event Perception in Mobile Interaction

In this paper, we explore how people perceive interactive activities in order to inform navigation history design on mobile devices. Following event segmentation method, we asked 12 participants to break six episodes of mobile interaction into segments, organize the segments, and identify those deemed representative. Three findings emerged. Firstly, when making sense of mobile interaction, users concentrate on the content objects on which actions are performed. This indicates the value of content-centric designs in navigation history and other mobile UI designs. The content objects are data objects and their collections meaningful to the person dealing with it, for example, photos, messages, or albums. Secondly, users tend to employ two-level hierarchies in grouping segments, and use the similarity in content objects and applications as a reference. They deem the segments as representative where objects are created or changed, or where sharing or querying acts take place. These findings indicate how a navigation history design should organize and prioritize mobile interaction events. Finally, event perception shows relatively low inter-participant consensus, which indicates that navigation history designs have to accommodate large individual differences. Event Perception in 3 Event Perception in Mobile Interaction: Toward Better Navigation History Design on Mobile Devices Imagine seeing somebody using a mobile Web browser intensively to find information about, say, flight departures from the Boston airport. The episode would probably take somewhere between 10 seconds and 10 minutes to complete, during which you would see the user clicking links, typing information into search fields, clicking buttons, scrolling up and down, etc. Now, another observer could perceive exactly the same events in some other way, perhaps in terms of navigation, querying, browsing, or searching for information. Understanding how people perceive interactive activities could be useful input for navigation history design. A design of navigation history typically collects, organizes, and presents past views or a summary of past views for users to continue or repeat their tasks (e.g. Heer, Mackinlay, Stolte, & Agrawala, 2008; Kaasten, Greenberg, & Edwards, 2001; Tang, Lin, Pierce, Whittaker, & Drews, 2007; Tauscher & Greenberg, 1997; Won, Jin, & Hong, 2009), or to navigate in a user interface (UI) (e.g. Park & Kim, 2000). If a design follows the principle of how people perceive their activities, the design has better chance to be well understood and used. Event perception, the psychological branch exploring these cognitive principles, however, rarely addresses interactive activities. Find a review on event perception in (Zacks, 2008; Zacks & Tversky, 2001). This study aims to fill the gap. It was conducted as input to designed history mechanisms in the Linked Internet UI concept (Cui, Honkala, Pihkala, Kinnunen, & Grassel, 2010). In this study, we explore event perception in mobile interactions, focusing on three key research questions. Question 1: What are the natural units of interaction for users? For example, the Web page or its counterparts might be a natural unit in a navigation history design, but we do Event Perception in 4 not know whether users perceive events on the basis of interface elements or something else. Question 2: How do users prioritize and group elements of a perceived episode of mobile interaction? These cognitive rules reveal the key events and their structures that a navigation history design needs to focus on. Question 3: To what extent do users agree with each other in their perceptions? A universally usable interaction demands reasonable consensus, or all users mostly segment and organize perception of a given mobile event in similar ways. We used event segmentation, a common method in event perception, in this study. The method usually involves asking observers to segment films of events, such as a person making a bed or assembling a saxophone (Newtson, 1973). Applying it to our context, we asked 12 participants to annotate videos of someone using a mobile device to complete everyday tasks. They had to identify units in the videos, organize the units into structures, identify the units deemed to represent each video well, and verbally explain their logic. The contribution of this paper is threefold. To our knowledge, the study is one of the first empirical studies that explore principles of how users understand mobile interactions. It is driven by a design purpose, and its findings are interpreted in the context of navigation history design. Finally, the study introduces several methods to calculate the extent to which the participants agree with each other, which can apply to all types of event perception studies. The paper is organized as follows: We start with a literature review highlighting some key research on event perception. After that, we explain our research method and the result of our study. Finally, we discuss the research and design implications in the end of the paper. Event Perception in 5 Related Work Event Perception Event perception researchers explore “set of cognitive mechanisms by which observers pick out meaningful spatiotemporal wholes from the stream of experience, recognize them, and identify their characteristics” (Zacks, 2008). One common method employed is to ask observers to segment footage of events into units as they watch them, or the event segmentation technique. The breakpoints – reported by observers when they perceived one unit ending and another beginning – are analyzed to reveal event perception principles (Newtson, 1973). A recurring finding has been that people naturally perceive goal-oriented activities as consisting of discrete parts and subparts (Hard, Lozano, & Tversky, 2006; Kurby & Zacks, 2008; Zacks, Tversky, & Iyer, 2001). For example, “making a bed” is a continuous activity but may be perceived as consisting of “putting on the sheet” and “putting the pillows in their cases.” “Putting on the sheet,” as a part, has its subparts, such as “unfolding the sheet,” “spreading it out,” and “straightening it out.” There is evidence that observers parse an activity into parts on the basis of distinctive sensory factors in a bottom-up process (Zacks, 2004) and proceed from their understanding of actors’ goals in a top-down process (Hard et al., 2006). Most likely, the sensory factors and conceptual interpretations are correlated: when one goal is completed and pursuit of another initiated, there are also physical changes, and vice versa (Kurby & Zacks, 2008). Some studies suggest a hierarchical structure of event perception: coarse-level, using the largest units, and fine-level, using the smallest natural units (Hard et al., 2006; Zacks et al., 2001), with, occasionally, a medium-level, using the granularity in between (Iqbal & Bailey, 2007). Coarse-grained structural breakpoints typically represent the introduction of objects and broad actions applied to them, while the breakpoints in fine-grained perception are more precise Event Perception in 6 actions applied to the same objects. The two kinds of breakpoints coincide at the same locations significantly more often than could occur by chance alone, which indicates that event perception indeed follows a hierarchical structure (Zacks et al., 2001). According to common coding theory, the same schemata underlie the perception of others’ actions and the planning of one’s own actions (Hommel, Muesseler, Aschersleben, & Prinz, 2001). This implies that people will apply a similar hierarchical structure whether they are perceiving the event online, recalling it after watching it, or even imagining the event from telegraphic descriptions (Zacks et al., 2001). Interactive Activities The findings from event perception studies have been applied to the field of human– computer interaction (HCI). The finding that an event is perceived as a structure of discrete units has been used to explain why animations have not yielded better performance than their equivalent static diagrams have in various contexts (Tversky, Morrison, & Betrancourt, 2002). The finding that a breakpoint reflects internal transitions in perception or cognition between two meaningful task units has been used to infer an opportune time for presenting interrupting notifications (Adamczyk & Bailey, 2004; Iqbal & Bailey, 2007). The finding that a breakpoint is typically characterized by a detectable change in sensory stimulus has been used to enhance computer vision work (Rui & Anandan, 2000). In contrast with the popularity of research on applying event perception principles, there are few empirical studies on exploring the principles of interactive activities themselves. In one exceptional case, Adamczyk and Bailey (2004) used the event segmentation technique to elicit task models in order to infer optimal interruption times. They asked 25 participants to segment three videos (of document editing, video watching, and Web searching) through breakpoints at coarse and fine levels. The researchers confirmed that observers can break an interaction Event Perception in 7 sequence into a hierarchical structure, and they suggested weighing the potential interruption cost at each breakpoint by measuring how much the users agree with each other. A follow-up study was done by Iqbal and Bailey (2007), who asked 24 participants to segment three videos (of document editing, image manipulation, and programming) at coarse, medium, and fine level. The results showed that coarseand medium-level breakpoints are associated with switching between applications; coarse-level breakpoints indicate switches in unrelated applications, whereas medium-level breakpoints indicate switches to related applications. Fine-level breakpoints were tied to actions applied to the content within an application – for example, completing formatting commands, searches, and copy-and-paste sequences in a documen

[1]  Karl Gyllstrom Passages through time: chronicling users' information interaction history by recording when and what they read , 2009, IUI.

[2]  DIMITRIOS PIERRAKOS,et al.  User Modeling and User-Adapted Interaction , 1994, User Modeling and User-Adapted Interaction.

[3]  Jaime Teevan,et al.  Visual snippets: summarizing web pages for search and revisitation , 2009, CHI.

[4]  Kerry Rodden,et al.  Smartback: supporting users in back navigation , 2004, WWW '04.

[5]  Carl Gutwin,et al.  Revisiting read wear: analysis, design, and evaluation of a footprints scrollbar , 2009, CHI.

[6]  Virpi Roto,et al.  How people use the web on mobile devices , 2008, WWW.

[7]  Jeffrey M. Zacks,et al.  Event perception , 2011, Scholarpedia.

[8]  Mikko Honkala,et al.  Linked internet UI: a mobile user interface optimized for social networking , 2010, Mobile HCI.

[9]  Virpi Roto,et al.  Graphical History List with Multi-window Support on a Mobile Web Browser , 2008, 2008 Third International Conference on Internet and Web Applications and Services.

[10]  Brian P. Bailey,et al.  If not now, when?: the effects of interruption at different moments within task execution , 2004, CHI.

[11]  James D. Hollan,et al.  Edit wear and read wear , 1992, CHI.

[12]  John C. Tang,et al.  Recent shortcuts: using recent interactions to support shared activities , 2007, CHI.

[13]  Saul Greenberg,et al.  Issues of Page Representation and Organisation in Web Browser's Revisitation Tools , 2000, Australas. J. Inf. Syst..

[14]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[15]  G. Aschersleben,et al.  The Theory of Event Coding (TEC): a framework for perception and action planning. , 2001, The Behavioral and brain sciences.

[16]  Barbara Tversky,et al.  Animation: can it facilitate? , 2002, Int. J. Hum. Comput. Stud..

[17]  Susan H. Gray,et al.  Using protocol analyses and drawings to study mental model construction during hypertext navigation , 1990, Int. J. Hum. Comput. Interact..

[18]  Brian P. Bailey,et al.  Understanding and Developing Models for Detecting and Differentiating Breakpoints during Task Execution , 2006 .

[19]  Mary Czerwinski,et al.  An Investigation of Memory for Daily Computing Events , 2002 .

[20]  Michael D. Byrne,et al.  The tangled Web we wove: a taskonomy of WWW use , 1999, CHI '99.

[21]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[22]  Soren Lauesen User Interface Design: A Software Engineering Perspective , 2004 .

[23]  Jason I. Hong,et al.  Contextual web history: using visual and contextual cues to improve web browser history , 2009, CHI.

[24]  Susan T. Dumais,et al.  Fast, flexible filtering with phlat , 2006, CHI.

[25]  Hannu Verkasalo,et al.  Handset-based analysis of mobile service usage , 2009 .

[26]  Barry Smyth,et al.  Understanding the intent behind mobile information needs , 2009, IUI.

[27]  Jeffrey M. Zacks,et al.  Using movement and intentions to understand simple events , 2004, Cogn. Sci..

[28]  Carol A. Taylor,et al.  A framework for understanding mobile internet motivations and behaviors , 2008, CHI Extended Abstracts.

[29]  James D. Hollan,et al.  A diary study of mobile information needs , 2008, CHI.

[30]  Saul Greenberg,et al.  How people revisit web pages: empirical findings and implications for the design of history systems , 1997, Int. J. Hum. Comput. Stud..

[31]  Darren Newtson Attribution and the unit of perception of ongoing behavior. , 1973 .

[32]  Eelco Herder,et al.  Web page revisitation revisited: implications of a long-term click-stream study of browser usage , 2007, CHI.

[33]  B. Tversky,et al.  Hierarchical encoding of behavior: translating perception into action. , 2006, Journal of experimental psychology. General.

[34]  Jeffrey M. Zacks,et al.  Perceiving, remembering, and communicating structure in events. , 2001, Journal of experimental psychology. General.

[35]  Saul Greenberg,et al.  Integrating back, history and bookmarks in web browsers , 2001, CHI Extended Abstracts.

[36]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[37]  Brian P. Bailey,et al.  Understanding and developing models for detecting and differentiating breakpoints during interactive tasks , 2007, CHI.

[38]  Stephen C. Hirtle,et al.  The Nature of Landmarks for Real and Electronic Spaces , 1999, COSIT.

[39]  Jeffrey M. Zacks,et al.  Segmentation in the perception and memory of events , 2008, Trends in Cognitive Sciences.

[40]  Fabio Pianesi,et al.  The influence of personality factors on visitor attitudes towards adaptivity dimensions for mobile museum guides , 2006, User Modeling and User-Adapted Interaction.

[41]  Jinwoo Kim,et al.  Contextual Navigation Aids for Two World Wide Web Systems , 2000, Int. J. Hum. Comput. Interact..

[42]  Saul Greenberg,et al.  How People Recognise Previously Seen Web Pages from Titles, URLs and Thumbnails , 2001 .

[43]  Pertti Huuskonen,et al.  Personal Content Experience: Managing Digital Life in the Mobile Age , 2007 .

[44]  Yong Rui,et al.  Segmenting visual actions based on spatio-temporal motion patterns , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[45]  Jeffrey Heer,et al.  Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation , 2008, IEEE Transactions on Visualization and Computer Graphics.