Comprehension of business process models: Insight into cognitive strategies via eye tracking

Process Models (PM) are visual documentations of the business processes within or across enterprises. Activities (tasks) are arranged together into a model (i.e., similar to flowcharts). This study aimed at understanding the underlying structure of PM comprehension. Though standards for describing PM have been defined, the cognitive work load they evoke, their structure, and the efficacy of information transmission are only partially understood. Two studies were conducted to better differentiate the concept of visual literacy (VL) and logical reasoning in interpreting PM. Study I: A total of 1047 students from 52 school classes were assessed. Three different process models of increasing complexity were presented on tablets. Additionally, written labels of the models’ elements were randomly allocated to scholars in a 3-group between-subjects design. Comprehension of process models was assessed by a series of 3 × 4 (=12) dichotomous test items. Latent Class Analysis of solved items revealed 6 qualitatively differing solution patterns, suggesting that a single test score is insufficient to reflect participants’ performance. Study II: Overall, 21 experts and 15 novices with respect to visual literacy were presented the same set of PMs as in Study I, while wearing eye tracking glasses. The fixation duration on relevant parts of the PM and on questions were recorded, as well as the total time needed to solve all 12 test items. The number of gaze transitions between process model and comprehension questions was measured as well. Being an expert in visual literacy did not alter the capability of correctly understanding graphical logical PMs. Presenting PMs that are labelled by single letters had a significant influence on reducing the time spent on irrelevant model parts but did not affect the fixation duration on relevant areas of interest. Both samples’ participants required longer response times with increasing model complexity. The number of toggles (i.e., gaze transitions between model and statement area of interest) was predictive for membership in one of the latent classes. Contrary to expectations, denoting the PM events and decisions not with real-world descriptions, but with single letters, led to lower cognitive workload in responding to comprehension questions and to better results. Visual Literacy experts could neither outperform novices nor high-school students in comprehending PM.

[1]  Manfred Reichert,et al.  Using Insights from Cognitive Neuroscience to Investigate the Effects of Event-Driven Process Chains on Process Model Comprehension , 2017, Business Process Management Workshops.

[2]  Jürgen Rost,et al.  Test Theory with Qualitative and Quantitative Latent Variables , 1988 .

[3]  J. Recker,et al.  Does It Matter Which Process Modelling Language We Teach or Use? An Experimental Study on Understanding Process Modelling Languages without Formal Education , 2007 .

[4]  Manfred Reichert,et al.  Cognitive Insights into Business Process Model Comprehension: Preliminary Results for Experienced and Inexperienced Individuals , 2017, BPMDS/EMMSAD@CAiSE.

[5]  Jorge Munoz-Gama,et al.  Process mining in healthcare: A literature review , 2016, J. Biomed. Informatics.

[6]  Manfred Reichert,et al.  Utilizing the Capabilities Offered by Eye-Tracking to Foster Novices' Comprehension of Business Process Models , 2018, ICCC.

[7]  Marie-France Barthet,et al.  What kind of interface for expert systems , 1991 .

[8]  E. Reingold,et al.  Eye movements and visual expertise in chess and medicine , 2011 .

[9]  Ruth Sara Aguilar-Savén,et al.  Business process modelling: Review and framework , 2004 .

[10]  Hans-Jürgen Bucher,et al.  The relevance of attention for selecting news content. An eye-tracking study on attention patterns in the reception of print and online media , 2006 .

[11]  Giuseppe Scanniello,et al.  On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments , 2013, Software & Systems Modeling.

[12]  Andreas Gegenfurtner,et al.  Effects of eye movement modeling examples on adaptive expertise in medical image diagnosis , 2017, Comput. Educ..

[13]  V. Neuhaus,et al.  Latent Class Analysis , 2010 .

[14]  Martin Meißner,et al.  The Promise of Eye-Tracking Methodology in Organizational Research: A Taxonomy, Review, and Future Avenues , 2019 .

[15]  Markus Nüttgens,et al.  Eye Tracking Experiments in Business Process Modeling: Agenda Setting and Proof of Concept , 2011, EMISA.

[16]  Wilson S. Geisler,et al.  Optimal eye movement strategies in visual search , 2005, Nature.

[17]  H. Gruber,et al.  Eye tracking in Educational Science: Theoretical frameworks and research agendas. , 2017, Journal of eye movement research.

[18]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[19]  Romain Martin,et al.  Eye Movements and Cognitive Strategy in a Fluid Intelligence Test: Item Type Analysis , 2018, Front. Psychol..

[20]  R. Mayer.  Learning strategies for making sense out of expository text: The SOI model for guiding three cognitive processes in knowledge construction , 1996 .

[21]  Rune Pettersson,et al.  Toward a Cohesive Theory of Visual Literacy , 2011 .

[22]  Markku Tukiainen,et al.  An eye-tracking methodology for characterizing program comprehension processes , 2006, ETRA.

[23]  E. Brumberger,et al.  Visual Literacy and the Digital Native: An Examination of the MillennialLearner , 2011 .

[24]  Louis A. Schultheiss,et al.  Techniques of flow-charting , 1963 .

[25]  N. Charness,et al.  The perceptual aspect of skilled performance in chess: Evidence from eye movements , 2001, Memory & cognition.

[26]  Markus Sommer,et al.  Reducing response elimination strategies enhances the construct validity of figural matrices , 2013 .

[27]  Eyal M. Reingold,et al.  Expert vs. novice differences in the detection of relevant information during a chess game: evidence from eye movements , 2014, Front. Psychol..

[28]  Ornella Robutti,et al.  How students read mathematical representations: An eye tracking study , 2009 .

[29]  J. Sweller.  Implications of Cognitive Load Theory for Multimedia Learning , 2005, The Cambridge Handbook of Multimedia Learning.

[30]  Rainer Koschke,et al.  How do professional developers comprehend software? , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[31]  N. Charness,et al.  Visual Span in Expert Chess Players: Evidence From Eye Movements , 2001, Psychological science.

[32]  Jean-Daniel Fekete,et al.  A Principled Way of Assessing Visualization Literacy , 2014, IEEE Transactions on Visualization and Computer Graphics.

[33]  Silvia Wen-Yu Lee,et al.  A review of using eye-tracking technology in exploring learning from 2000 to 2012 , 2013 .

[34]  Jörg Becker,et al.  Guidelines of Business Process Modeling , 2000, Business Process Management.

[35]  V. Shute.  Focus on Formative Feedback , 2007 .

[36]  J. Rost,et al.  Applications of Latent Trait and Latent Class Models in the Social Sciences , 1998 .

[37]  J. Sweller,et al.  Natural Information Processing Systems , 2006 .

[38]  Kenneth Holmqvist,et al.  Threats to the validity of eye-movement research in psychology , 2018, Behavior research methods.

[39]  E. Vakil,et al.  Solving the Raven Progressive Matrices by adults with intellectual disability with/without Down syndrome: different cognitive patterns as indicated by eye-movements. , 2012, Research in developmental disabilities.

[40]  Jan Recker,et al.  The Effects of Content Presentation Format and User Characteristics on Novice Developers' Understanding of Process Models , 2011, Commun. Assoc. Inf. Syst..

[41]  Fang-Ying Yang,et al.  PROBING THE RELATIONSHIP BETWEEN PROCESS OF SPATIAL PROBLEMS SOLVING AND SCIENCE LEARNING: AN EYE TRACKING APPROACH , 2014 .

[42]  C. Mitchell Dayton,et al.  13 Latent Class Analysis in Psychometrics , 2006 .

[43]  Volker Gruhn,et al.  Adopting the Cognitive Complexity Measure for Business Process Models , 2006, 2006 5th IEEE International Conference on Cognitive Informatics.

[44]  Mark von Rosing,et al.  Business Process Model and Notation - BPMN , 2015, The Complete Business Process Handbook, Vol. I.

[45]  Mei-Hui Su,et al.  Characterization of scientific workflows , 2008, 2008 Third Workshop on Workflows in Support of Large-Scale Science.

[46]  Jennifer M. Brill,et al.  Visual Literacy Defined – The Results of a Delphi Study: Can IVLA (Operationally) Define Visual Literacy? , 2007 .

[47]  Matthieu Lenoir,et al.  Measuring dwell time percentage from head-mounted eye-tracking data – comparison of a frame-by-frame and a fixation-by-fixation analysis , 2015, Ergonomics.

[48]  C. E. Bethell-Fox,et al.  Adaptive reasoning: Componential and eye movement analysis of geometric analogy performance ☆ , 1984 .

[49]  Mustafa C. Ungan.  Towards a better understanding of process documentation , 2006 .

[50]  R. Säljö,et al.  Expertise Differences in the Comprehension of Visualizations: a Meta-Analysis of Eye-Tracking Research in Professional Domains , 2011 .

[51]  Jan Mendling,et al.  Understanding Business Process Models: The Costs and Benefits of Structuredness , 2012, CAiSE.

[52]  Svein Magnussen,et al.  Expertise in Pictorial Perception: Eye-Movement Patterns and Visual Memory in Artists and Laymen , 2007, Perception.

[53]  Manfred Reichert,et al.  Eye Tracking Experiments on Process Model Comprehension: Lessons Learned , 2017, BPMDS/EMMSAD@CAiSE.

[54]  Jan Mendling,et al.  Eye-Tracking the Factors of Process Model Comprehension Tasks , 2013, CAiSE.

[55]  Kevin Andrews,et al.  A Smart Mobile Assessment Tool for Collecting Data in Large-Scale Educational Studies , 2018, FNC/MobiSPC.

[56]  Isabell Wartenburger,et al.  Resource allocation and fluid intelligence: insights from pupillometry. , 2010, Psychophysiology.

[57]  Mark Strembeck,et al.  The Influence of Notational Deficiencies on Process Model Comprehension , 2013, J. Assoc. Inf. Syst..

[58]  John Sweller,et al.  Measuring cognitive load , 2018, Perspectives on Medical Education.

[59]  Adam Fadlalla,et al.  Business information visualization intellectual contributions: An integrative framework of visualization capabilities and dimensions of visual intelligence , 2016, Decis. Support Syst..

[60]  Kathrin Figl.  Comprehension of Procedural Visual Business Process Models , 2017, Bus. Inf. Syst. Eng..

[61]  Guttorm Sindre,et al.  Evaluating the Quality of Process Models: Empirical Testing of a Quality Framework , 2002, ER.

[62]  E. B. Andersen,et al.  A goodness of fit test for the rasch model , 1973 .

[63]  J. Sweller,et al.  Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions , 2005 .

[64]  Mark Strembeck,et al.  Factors of process model comprehension - Findings from a series of experiments , 2012, Decis. Support Syst..

[65]  Taylor R. Hayes,et al.  Mapping and correcting the influence of gaze position on pupil size measurements , 2015, Behavior Research Methods.

[66]  Per B Sederberg,et al.  A novel method for analyzing sequential eye movements reveals strategic influence on Raven's Advanced Progressive Matrices. , 2011, Journal of vision.