Process-mining enabled feedback: "Tell me what I did wrong" vs. "tell me how to do it right"

Fast advancement of technology has led to an increased interest for using information technology to provide feedback based on learning behavior observations. This work outlines a novel approach for analyzing behavioral learner data through the application of process mining techniques specifically targeting a complex problem solving process. We realize this in the context of one particular learning case, namely, domain modeling. This work extends our previous research on process-mining analysis of domain modeling behavior of novices by elaborating with new insights from a replication study enhanced with an extra observation on how novices verify/validate models. The findings include a set of typical modeling and validation patterns that can be used to improve teaching guidance for domain modeling courses. From a scientific viewpoint, the results contribute to improving our knowledge on the cognitive aspects of problem-solving behavior of novices in the area of domain modeling, specifically regarding process-oriented feedback as opposed to traditional outcome feedback (is a solution correct? Why (not)?) usually applied in this type of courses. Ultimately, the outcomes of the work can be inspirational outside of the area of domain modeling as learning event data is becoming readily available through virtual learning environments and other information systems. This work extends our previous research on process mining of learner data.Complex task solving process for conceptual modeling by novices is analized.The extension includes an observation of extra activities for model validation.Modeling/validation patterns indicative for learning outcomes are found.The findings provide a basis for process-oriented guidance and feedback.

[1]  Mario Piattini,et al.  Using Practitioners for Assessing the Understandability of UML Statechart Diagrams with Composite States , 2007, ER Workshops.

[2]  Keng Siau,et al.  Can UML Be Simplified? Practitioner Use of UML in Separate Domains , 2007, EMMSAD.

[3]  Jan Recker,et al.  How Novices Model Business Processes , 2010, BPM.

[4]  Arne Sølvberg,et al.  Understanding quality in conceptual modeling , 1994, IEEE Software.

[5]  Bernhard Thalheim,et al.  Handbook of Conceptual Modeling - Theory, Practice, and Research Challenges , 2011 .

[6]  Gabriele Bavota,et al.  Identifying the weaknesses of UML class diagrams during data model comprehension , 2011, MODELS'11.

[7]  Mario Piattini,et al.  Using Controlled Experiments for Validating UML Statechart Diagrams Measures , 2007, IWSM/Mensura.

[8]  Mykola Pechenizkiy,et al.  Process mining from educational data (Chapter 9) , 2011, EDM 2011.

[9]  P. Winne,et al.  Feedback and Self-Regulated Learning: A Theoretical Synthesis , 1995 .

[10]  Florian Matthes,et al.  A Meta-language for EA Information Modeling - State-of-the-Art and Requirements Elicitation , 2010, BMMDS/EMMSAD.

[11]  KENG SIAU.  Unified Modeling Language ( UML ) — A Complexity Analysis , 2017 .

[12]  Ravi Paul,et al.  Analyzing the structure of expert knowledge , 2006, Inf. Manag..

[13]  P. Reimann,et al.  Process mining techniques for analysing patterns and strategies in students’ self-regulated learning , 2013, Metacognition and Learning.

[14]  Patrick Wessa,et al.  A constructivist approach in a blended e-learning environment for statistics , 2015, Interact. Learn. Environ..

[15]  Monique Snoeck,et al.  Lightweight Semantic Prototyper for Conceptual Modeling , 2014, ER Workshops.

[16]  S. Järvelä,et al.  Combining Individual and Group-Level Perspectives for Studying Collaborative Knowledge Construction in Context. , 2007 .

[17]  Monique Snoeck,et al.  Technology-Enhanced Support for Learning Conceptual Modeling , 2012, BMMDS/EMMSAD.

[18]  Baruch B. Schwarz,et al.  Outcome feedback during collaborative learning: Contingencies between feedback and dyad composition , 2014 .

[19]  Stijn Hoppenbrouwers,et al.  Cognitive Mechanisms of Conceptual Modelling - How Do People Do It? , 2013, ER.

[20]  Kalina Yacef,et al.  Process Mining to Support Students' Collaborative Writing , 2010, EDM.

[21]  B. Zimmerman.  Self-Regulated Learning and Academic Achievement: An Overview , 1990 .

[22]  Barbara Weber,et al.  The Modeling Mind: Behavior Patterns in Process Modeling , 2014, BMMDS/EMMSAD.

[23]  Jochen De Weerdt,et al.  Process Mining for the multi-faceted analysis of business processes - A case study in a financial services organization , 2013, Comput. Ind..

[24]  Monique Snoeck,et al.  Assessing the effectiveness of feedback enabled simulation in teaching conceptual modeling , 2014, Comput. Educ..

[25]  Nicholas P. Vitalari,et al.  Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do? , 1998, J. Manag. Inf. Syst..

[26]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[27]  D. DavisFred,et al.  User Acceptance of Computer Technology , 1989 .

[28]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[29]  Mykola Pechenizkiy,et al.  Process Mining Online Assessment Data , 2009, EDM.

[30]  Wil M. P. van der Aalst,et al.  Learning Analytics on Coursera Event Data: A Process Mining Approach , 2015, SIMPDA.

[31]  R. Keith Sawyer,et al.  Foundations of the learning sciences , 2014 .

[32]  John A. Zachman,et al.  A Framework for Information Systems Architecture , 1987, IBM Syst. J..

[33]  S. Narciss.  Feedback Strategies for Interactive Learning Tasks , 2007 .

[34]  Mario Piattini,et al.  Assessing the understandability of UML statechart diagrams with composite states—A family of empirical studies , 2009, Empirical Software Engineering.

[35]  Monique Snoeck,et al.  Do we need to teach testing skills in courses on requirements engineering and modelling , 2014 .

[36]  James Martin.  Information engineering, book ii planning and analysis. , 2014 .

[37]  Mario Piattini,et al.  The impact of structural complexity on the understandability of UML statechart diagrams , 2010, Inf. Sci..

[38]  Deborah Compeau,et al.  Computer Self-Efficacy: Development of a Measure and Initial Test , 1995, MIS Q..

[39]  Jan Mendling,et al.  Styles in business process modeling: an exploration and a model , 2015, Software & Systems Modeling.

[40]  Carl-Henric Nilsson,et al.  Metacognition and Learning Journals in Higher Education , 2013 .

[41]  Monique Snoeck,et al.  Enterprise Information Systems Engineering : The MERODE Approach , 2014 .

[42]  Monique Snoeck,et al.  Enterprise architecture for small and medium-sized enterprises: a starting point for bringing EA to SMEs, based on adoption models , 2012, CENTERIS 2012.

[43]  Klaus Pohl,et al.  Requirements Engineering for Embedded Systems: An Investigation of Industry Needs , 2011, REFSQ.

[44]  Mario Piattini,et al.  Defining and Validating Metrics for UML Statechart Diagrams , 2002 .

[45]  Jan Mendling,et al.  Tying Process Model Quality to the Modeling Process: The Impact of Structuring, Movement, and Speed , 2012, BPM.

[46]  Christina Keller,et al.  User Acceptance of Virtual Learning Environments: A Case Study from Three Northern European Universities , 2009, Commun. Assoc. Inf. Syst..

[47]  Monique Snoeck,et al.  Effects of Simulation on Novices' Understanding of the Concept of Inheritance in Conceptual Modeling , 2015, ER Workshops.

[48]  Mykola Pechenizkiy,et al.  From Local Patterns to Global Models: Towards Domain Driven Educational Process Mining , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[49]  Monique Snoeck,et al.  Process mining analysis of conceptual modeling behavior of novices - empirical study using JMermaid modeling and experimental logging environment , 2014, Comput. Hum. Behav..

[50]  Keng Siau,et al.  Unified Modeling Language: A Complexity Analysis , 2001, J. Database Manag..

[51]  Keng Siau,et al.  Theoretical vs. Practical Complexity: The Case of UML , 2005, J. Database Manag..

[52]  William A. Wallace,et al.  Visualization and the process of modeling: a cognitive-theoretic view , 2000, KDD '00.

[53]  Monique Snoeck,et al.  Feedback-Enabled MDA-Prototyping Effects on Modeling Knowledge , 2013, BMMDS/EMMSAD.

[54]  Giuseppe Santucci,et al.  Serious : a UML based metric for effort estimation , 2002 .

[55]  Wilhelmiina Hämäläinen,et al.  Classiers for Educational Data Mining , 2010 .

[56]  V. Shute.  Focus on Formative Feedback , 2008 .

[57]  Geert Poels,et al.  A visual analysis of the process of process modeling , 2014, Information Systems and e-Business Management.

[58]  Boudewijn F. van Dongen,et al.  ProM: The Process Mining Toolkit , 2009, BPM.

[59]  S. Järvelä,et al.  Sequential and temporal characteristics of self and socially regulated learning , 2014 .

[60]  Jean Bézivin,et al.  Model Driven Engineering: An Emerging Technical Space , 2005, GTTSE.

[61]  Filippo Ricca,et al.  What are the used UML diagrams? A Preliminary Survey , 2013, EESSMOD@MoDELS.

[62]  José Javier Dolado,et al.  Evaluation of the comprehension of the dynamic modeling in UML , 2004, Inf. Softw. Technol..

[63]  Patrik Pluchino,et al.  Eye-movement modeling of integrative reading of an illustrated text: Effects on processing and learning , 2015 .

[64]  Remigijus Gustas,et al.  Conceptual Modeling and Integration of Static and Dynamic Aspects of Service Architectures , 2010, ONTOSE.

[65]  Mario Piattini,et al.  A conceptual modeling quality framework , 2011, Software Quality Journal.

[66]  John P. Buerck,et al.  Facilitating Self-Regulated Learning With Technology , 2011 .

[67]  Mario Piattini,et al.  Defining Metrics for UML Statechart Diagrams in a Methodological Way , 2003, ER.

[68]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[69]  Monique Snoeck,et al.  A PIM-to-Code Requirements Engineering Framework , 2013, MODELSWARD.

[70]  S. Järvelä,et al.  Self-Regulated, Co-Regulated, and Socially Shared Regulation of Learning , 2011 .

[71]  E. Tansley,et al.  Using ontology to validate conceptual models , 2003, CACM.

[72]  Monique Snoeck,et al.  Consistency by Construction: The Case of MERODE , 2003, ER.