A Simulation Model of Intermittently Controlled Point-and-Click Behaviour

We present a novel simulation model of point-and-click behaviour that is applicable both when a target is stationary or moving. To enable more realistic simulation than existing models, the model proposed in this study takes into account key features of the user and the external environment, such as intermittent motor control, click decision-making, visual perception, upper limb kinematics and the effect of input device. The simulated user’s point-and-click behaviour is formulated as a Markov decision process (MDP), and the user’s policy of action is optimised through deep reinforcement learning. As a result, our model successfully and accurately reproduced the trial completion time, distribution of click endpoints, and cursor trajectories of real users. Through an ablation study, we showed how the simulation results change when the model’s sub-modules are individually removed. The implemented model and dataset are publicly available.

[1]  Antti Oulasvirta,et al.  MenuOptimizer: interactive optimization of menu systems , 2013, UIST.

[2]  Antti Oulasvirta,et al.  Modelling Learning of New Keyboard Layouts , 2017, CHI.

[3]  Byungjoo Lee,et al.  An Intermittent Click Planning Model , 2020, CHI.

[4]  Makoto Sato,et al.  Learning and generation of goal-directed arm reaching from scratch , 2009, Neural Networks.

[5]  Antti Oulasvirta,et al.  Modelling Error Rates in Temporal Pointing , 2016, CHI.

[6]  Ezra S. Krendel,et al.  Mathematical Models of Human Pilot Behavior , 1974 .

[7]  A. Barto Learning to reach via corrective movements , 1999 .

[8]  R. Bellman A Markovian Decision Process , 1957 .

[9]  Shumin Zhai,et al.  Characterizing computer input with Fitts' law parameters-the information and non-information aspects of pointing , 2004, Int. J. Hum. Comput. Stud..

[10]  Antti Oulasvirta,et al.  AutoGain: Adapting Gain Functions by Optimizing Submovement Efficiency , 2016, ArXiv.

[11]  Shumin Zhai,et al.  Beyond Fitts' law: models for trajectory-based HCI tasks , 1997, CHI Extended Abstracts.

[12]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[13]  Robin T. Bye,et al.  The BUMP model of response planning: variable horizon predictive control accounts for the speed-accuracy tradeoffs and velocity profiles of aimed movement. , 2008, Human movement science.

[14]  A. Rangel,et al.  Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions , 2011, Proceedings of the National Academy of Sciences.

[15]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[16]  David E. Kieras,et al.  The GOMS family of user interface analysis techniques: comparison and contrast , 1996, TCHI.

[17]  Konrad Paul Kording,et al.  Review TRENDS in Cognitive Sciences Vol.10 No.7 July 2006 Special Issue: Probabilistic models of cognition Bayesian decision theory in sensorimotor control , 2022 .

[18]  Antti Oulasvirta,et al.  Optimal Sensor Position for a Computer Mouse , 2020, CHI.

[19]  Antti Oulasvirta,et al.  Neuromechanics of a Button Press , 2018, CHI.

[20]  Kris Luyten,et al.  Familiarisation: Restructuring Layouts with Visual Learning Models , 2018, IUI.

[21]  Shumin Zhai,et al.  FFitts law: modeling finger touch with fitts' law , 2013, CHI.

[22]  W. Jaschinski The Proximity-Fixation-Disparity Curve and the Preferred Viewing Distance at a Visual Display as an Indicator of Near Vision Fatigue , 2002, Optometry and vision science : official publication of the American Academy of Optometry.

[23]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[24]  Shumin Zhai,et al.  Predicting Finger-Touch Accuracy Based on the Dual Gaussian Distribution Model , 2016, UIST.

[25]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[26]  Denis Efimov,et al.  A switched dynamic model for pointing tasks with a computer mouse , 2019, Asian Journal of Control.

[27]  J. Tresilian Hitting a moving target: Perception and action in the timing of rapid interceptions , 2005, Perception & psychophysics.

[28]  Antti Oulasvirta,et al.  Combinatorial Optimization of Graphical User Interface Designs , 2020, Proceedings of the IEEE.

[29]  Antti Oulasvirta,et al.  Sketchplore: Sketch and Explore with a Layout Optimiser , 2016, Conference on Designing Interactive Systems.

[30]  Myroslav Bachynskyi,et al.  Dynamics of Aimed Mid-air Movements , 2020, CHI.

[31]  Toshiyuki Kondo,et al.  Biological arm motion through reinforcement learning , 2004, Biological Cybernetics.

[32]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.

[33]  C J Worringham,et al.  Directional stimulus-response compatibility: a test of three alternative principles. , 1998, Ergonomics.

[34]  Robin T. Bye,et al.  The BUMP model of response planning: intermittent predictive control accounts for 10 Hz physiological tremor. , 2010, Human movement science.

[35]  Daniel Vogel,et al.  The Impact of Control-Display Gain on User Performance in Pointing Tasks , 2008, Hum. Comput. Interact..

[36]  W. H. Warren,et al.  Behavioral dynamics of intercepting a moving target , 2007, Experimental Brain Research.

[37]  Antti Oulasvirta,et al.  Moving Target Selection: A Cue Integration Model , 2018, CHI.

[38]  Jörg Müller,et al.  Control Theoretic Models of Pointing , 2017, ACM Trans. Comput. Hum. Interact..

[39]  Konrad Paul Kording,et al.  Sensory Cue Integration , 2011 .

[40]  Per Ola Kristensson,et al.  Improving two-thumb text entry on touchscreen devices , 2013, CHI.

[41]  Perttu Hämäläinen,et al.  Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning , 2020, CHI.

[42]  Lars Gruene,et al.  An Optimal Control Model of Mouse Pointing Using the LQR , 2020, ArXiv.

[43]  Antti Oulasvirta,et al.  Adaptive feature guidance: Modelling visual search with graphical layouts , 2020, Int. J. Hum. Comput. Stud..

[44]  I. Scott MacKenzie,et al.  Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts' law research in HCI , 2004, Int. J. Hum. Comput. Stud..

[45]  H. Zelaznik,et al.  Motor-output variability: a theory for the accuracy of rapid motor acts. , 1979, Psychological review.

[46]  Allen Newell,et al.  The model human processor: An engineering model of human performance. , 1986 .

[47]  Dario D. Salvucci An integrated model of eye movements and visual encoding , 2001, Cognitive Systems Research.

[48]  Eero P. Simoncelli,et al.  Noise characteristics and prior expectations in human visual speed perception , 2006, Nature Neuroscience.

[49]  Luc Florack Modeling Foveal Vision , 2007, SSVM.

[50]  David E. Kieras Using the Keystroke-Level Model to Estimate Execution Times , 2003 .

[51]  R M Church,et al.  Scalar Timing in Memory , 1984, Annals of the New York Academy of Sciences.

[52]  I. Scott MacKenzie,et al.  An error model for pointing based on Fitts' law , 2008, CHI.

[53]  A Diamond,et al.  Reaching control of a full-torso, modelled musculoskeletal robot using muscle synergies emergent under reinforcement learning , 2014, Bioinspiration & biomimetics.

[54]  Xiangmin Fan,et al.  Modeling the Uncertainty in 2D Moving Target Selection , 2019, UIST.

[55]  Byungjoo Lee,et al.  A Mouse With Two Optical Sensors That Eliminates Coordinate Disturbance During Skilled Strokes , 2015, Hum. Comput. Interact..

[56]  Antti Oulasvirta,et al.  Improvements to keyboard optimization with integer programming , 2014, UIST.

[57]  Shumin Zhai,et al.  Performance Optimization of Virtual Keyboards , 2002, Hum. Comput. Interact..

[58]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[59]  Zhou Wang,et al.  Video quality assessment using a statistical model of human visual speed perception. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[60]  Ray F. Lin,et al.  The use of ballistic movement as an additional method to assess performance of computer mice , 2015 .

[61]  Carl F. Craver,et al.  When mechanistic models explain , 2006, Synthese.

[62]  Nicolas Roussel,et al.  No more bricolage!: methods and tools to characterize, replicate and compare pointing transfer functions , 2011, UIST.

[63]  H. Cruse,et al.  On the cost functions for the control of the human arm movement , 1990, Biological Cybernetics.

[64]  M. C. Smith,et al.  Theories of the psychological refractory period. , 1967, Psychological bulletin.

[65]  Joohwan Kim,et al.  Latency of 30 ms Benefits First Person Targeting Tasks More Than Refresh Rate Above 60 Hz , 2019, SIGGRAPH Asia Technical Briefs.