Process mining based approach to performance evaluation in computer‐aided examinations

A considerable part of educational systems tends to be online and computer oriented. However, online examination may create some difficulties during the evaluation of student performance. Process mining which arises as a new concept presents various powerful techniques for processing and analyzing different types of data by making use of some advanced information technologies. This paper proposes a novel approach based on process mining for evaluating the performance of students that should follow certain tasks on the computer. The proposed approach is composed of two main phases which are process mining and similarity analysis. Automatic assessment is performed totally in six steps in order to obtain students’ final grades. In addition, cheat control is possible in the last step thanks to the similarity analysis. A real‐life application in an Enterprise Resource Planning (ERP) course is performed in order to present usefulness, validity and practicality of the proposed approach. Furthermore, to evaluate the performance of the assessment system, we compared the assessment mechanism against instructor. A total of 15 students’ answers belonging to computer‐aided exam are assessed by instructor and the results showed a very good agreement between the automatic assessment system and instructor.

[1]  Hajo A. Reijers,et al.  Process Mining on Databases: Unearthing Historical Data from Redo Logs , 2015, BPM.

[2]  Cw Christian Günther.  Process mining in flexible environments , 2009 .

[3]  Vallipuram Muthukkumarasamy,et al.  Performance analysis of GAME: A generic automated marking environment , 2008, Comput. Educ..

[4]  Alexander L. Wolf,et al.  Discovering models of software processes from event-based data , 1998, TSEM.

[5]  Boudewijn F. van Dongen,et al.  A Meta Model for Process Mining Data , 2005, EMOI-INTEROP.

[6]  K. Y. Tam,et al.  An operation sequence based similarity coefficient for part families formations , 1990 .

[7]  Christopher Douce,et al.  Automatic test-based assessment of programming: A review , 2005, JERC.

[8]  Jan Mendling,et al.  Bridging Abstraction Layers in Process Mining by Automated Matching of Events and Activities , 2013, BPM.

[9]  Jochen De Weerdt,et al.  Process Mining for the multi-faceted analysis of business processes - A case study in a financial services organization , 2013, Comput. Ind..

[10]  Matthew Fleming,et al.  A comparison of in-class and online quizzes on student exam performance , 2003, J. Comput. High. Educ..

[11]  Dimitrios Gunopulos,et al.  Mining Process Models from Workflow Logs , 1998, EDBT.

[12]  Femi Emmanuel Ayo,et al.  A probabilistic approach to event log completeness , 2017, Expert Syst. Appl..

[13]  Adil Baykasoğlu,et al.  MOCACEF 1.0: Multiple objective capability based approach to form part-machine groups for cellular manufacturing applications , 2000 .

[14]  Sheetal Kalra,et al.  Smart computing based student performance evaluation framework for engineering education , 2017, Comput. Appl. Eng. Educ..

[15]  Young Sik Kang,et al.  A Study of Process Mining-based Business Process Innovation☆ , 2016 .

[16]  Wil M. P. van der Aalst,et al.  Process Mining - Discovery, Conformance and Enhancement of Business Processes , 2011 .

[17]  Wil M.P. van der Aalst,et al.  Process Mining Put into Context , 2012, IEEE Internet Computing.

[18]  Angelina Prima Kurniati,et al.  Evaluation of the online assessment test using process mining (Case Study: Intensive English Center) , 2015, 2015 3rd International Conference on Information and Communication Technology (ICoICT).

[19]  Ioannis Hatzilygeroudis,et al.  An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance , 2017, International Journal of Artificial Intelligence in Education.

[20]  Wil M. P. van der Aalst,et al.  Fuzzy Mining - Adaptive Process Simplification Based on Multi-perspective Metrics , 2007, BPM.

[21]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[22]  Nathan Griffiths,et al.  The boss online submission and assessment system , 2005, JERC.

[23]  Wil M. P. van der Aalst,et al.  Process mining: a research agenda , 2004, Comput. Ind..

[24]  Goran Mausa,et al.  Using string similarity metrics for automated grading of SQL statements , 2015, 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[25]  Manuel Mucientes,et al.  Recompiling learning processes from event logs , 2016, Knowl. Based Syst..

[26]  Kyle L. Peck,et al.  An approach to designing computer-based evaluation of student constructed responses: Effects on achievement and instructional time , 1995, J. Comput. High. Educ..

[27]  Jorge Munoz-Gama,et al.  Process mining in healthcare: A literature review , 2016, J. Biomed. Informatics.

[28]  Jéan H. Greyling,et al.  Marking student programs using graph similarity , 2010, Comput. Educ..

[29]  David Jackson,et al.  Grading student programs using ASSYST , 1997, SIGCSE '97.

[30]  Michael T. Helmick.  Interface-based programming assignments and automatic grading of java programs , 2007, ITiCSE.

[31]  Hanim Maria Astuti,et al.  ANALYSIS OF CUSTOMER FULFILMENT WITH PROCESS MINING: A CASE STUDY IN A TELECOMMUNICATION COMPANY , 2015 .

[32]  Wil M. P. van der Aalst,et al.  Workflow Patterns , 2003, Distributed and Parallel Databases.

[33]  Xiaohong Su,et al.  Ability-training-oriented automated assessment in introductory programming course , 2011, Comput. Educ..

[34]  Viktor Kuncak,et al.  Software verification and graph similarity for automated evaluation of students' assignments , 2012, Inf. Softw. Technol..

[35]  Lauri Malmi,et al.  Visual Algorithm Simulation Exercise System with Automatic Assessment: TRAKLA2 , 2004, Informatics Educ..

[36]  Lale Özbakir,et al.  Genetic Programming Based Data Mining Approach to Dispatching Rule Selection in a Simulated Job Shop , 2010, Simul..

[37]  C. Humby,et al.  Process Mining: Data science in Action , 2014 .

[38]  Neil Smith,et al.  Automatically assessing graph‐based diagrams , 2008 .

[39]  Riyanarto Sarno,et al.  Hybrid Association Rule Learning and Process Mining for Fraud Detection , 2022 .

[40]  WangKuanquan,et al.  Ability-training-oriented automated assessment in introductory programming course , 2011 .

[41]  David Carrera,et al.  Business Process Mining from E-Commerce Web Logs , 2013, BPM.

[42]  Guido Schimm,et al.  Mining exact models of concurrent workflows , 2004, Comput. Ind..

[43]  Wondea Jung,et al.  The use of a process mining technique to characterize the work process of main control room crews: A feasibility study , 2016, Reliab. Eng. Syst. Saf..

[44]  Li Xiu,et al.  Application of data mining techniques in customer relationship management: A literature review and classification , 2009, Expert Syst. Appl..

[45]  José Luis Fernández Alemán.  Automated Assessment in a Programming Tools Course , 2011, IEEE Trans. Educ..

[46]  Ashutosh Tiwari,et al.  A review of business process mining: state-of-the-art and future trends , 2008, Bus. Process. Manag. J..

[47]  Imelda Atastina,et al.  Student registration process evaluation using process mining case study: IT Telkom , 2014, Ninth International Conference on Digital Information Management (ICDIM 2014).

[48]  Wil M. P. van der Aalst,et al.  Decomposing Process Mining Problems Using Passages , 2012, Petri Nets.

[49]  E. Smirnova,et al.  NEW TECHNOLOGY FOR PROGRAMMING TEACHING: PROCESS MINING USAGE , 2015 .

[50]  Boudewijn F. van Dongen,et al.  Workflow mining: A survey of issues and approaches , 2003, Data Knowl. Eng..

[51]  Wil M. P. van der Aalst,et al.  Business alignment: using process mining as a tool for Delta analysis and conformance testing , 2005, Requirements Engineering.

[52]  Athanasios Tsintsifas,et al.  Automated assessment and experiences of teaching programming , 2005, JERC.