MEEGA + : A Method for the Evaluation of Educational Games for Computing Education

Summarizes the paper under headings of background or context, objectives, method, main results, and conclusions. Introduction Problem Statement Indicates what the problem is; where is occurs, and who observe it. Research Objective Defines the evaluation using the formalized style used in the MEEGA+ model. Context Indicates environmental factors such as institution, course, and participants involved in the evaluation. Related Work How this research relates to existing research (studies)? Research Method Reports the methods used in the research, such as the MEEGA+ method (Petri et al., 2018), case studies (Yin, 2017; Wohlin et al., 2012), GQM (Basili et al., 1994). Evaluation Planning Object of study Indicates the game selected fort the evaluation. Evaluation goal Presents the defined evaluation goal and the analysis questions following the MEEGA+ model. Analyse the <name of the selected game> for the purpose of evaluate the quality in terms of usability and player experience from the students’ point of view in the context of higher computing education. Analysis questions AQ1: Does the <name of the evaluated game> has a good usability? AQ2: Does the <name of the evaluated game> provides a positive player experience? AQ3: How old are the students that compose the sample of the study? AQ4: What is the gender of the students that compose the sample of the study? AQ5: What is the frequency that the students play digital and/or non-digital games? Context details Indicates the place that the evaluation took place, such as institution and course. Research design Indicates the research design applied, following the definition of the MEEGA+ model. Case study design (one-shot post-test only). Schedule Indicates the schedule of the evaluation such as date and time. Number of Indicates the number of the approval provided by the Ethics INCoD – Brazilian Institute for Digital Convergence JULY 2018 40 the Ethics Committee approval Committee (if necessary). Execution Sample Description of the sample characteristics (demographic information). Preparation What has been done to prepare the execution of the evaluation (i.e., schedule, materials)? Game applied Indicates how game application took place and any deviations from plan. Data collection performed How data collection took place and any deviations from plan. Analysis Answer the analysis questions Summarizes the data collected and describes how it was analysed and answers each of the analysis questions defined. Game quality level Indicates the quality level of the evaluated game, obtained from the MEEGA+ scale. Discussion Evaluation of results Interprets and explains the findings from the Analysis section. Threats to validity Discusses the main threats to validity and mitigation strategies applied. Conclusions and Future Work Summary Provides a concise summary of the research objective and evaluation execution. Findings Identifies the most important results of the study. Improvement opportunities Suggestions for other studies to further investigate. Acknowledgements Identifies any sponsors, participants, and contributors who do not fulfil authorship criteria. References Lists all cited literature in the format requested by the publisher. Appendices Includes supplementary data and/or detailed analyses which might help others to use the results. INCoD – Brazilian Institute for Digital Convergence JULY 2018 41 4. Conclusions In this technical report, we presented the MEEGA+ method, an evolution of an evaluation model of educational games used as instructional strategy for computing education, improving the initial model proposed by Savi et al. (2011). The MEEGA+ method aims to provide a systematic support for the evaluation of games for computing education, focusing on the quality evaluation of educational games (including digital as well as non-digital games) in terms of usability and player experience. It is composed of an evaluation model (MEEGA+ model) defining quality aspects to evaluate a game, and a process (MEEGA+ process) guiding the conduction of the game evaluation. As next steps, we plan to continue conducting case studies evaluating games (digital and non-digital) for computing education using the MEEGA+ method in order to conduct a statistical analysis to confirm the decomposition of the MEEGA+ factors and dimensions. In addition, we plan evaluate the quality of the MEEGA+ method from the expert’s perspective. INCoD – Brazilian Institute for Digital Convergence JULY 2018 42 Acknowledgments We would like to thank all the students and instructors that accepted participate in the applications of the games using the MEEGA+ method. This work was supported by the CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico – www.cnpq.br), an entity of the Brazilian government focused on scientific and technological development. This work was partially conducted during a visiting scholar period at University of Cádiz, sponsored by the CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), Foundation within the Ministry of Education, Brazil (grant n. 88881.131485/2016-01). INCoD – Brazilian Institute for Digital Convergence JULY 2018 43 References Abt, C. C. (2002). Serious Games. Lanhan: University Press of America. ACM/IEEE-CS. (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science, 2013. Available on: <https://www.acm.org/education/CS2013final-report.pdf> Access: 06 Nov. 2017. Acuña, S. T.; Antonio, A.; Ferré, X.; López, M.; Maté, L. (2000). The Software process: modeling, evaluation and improvement. Handbook of Software Engineering and Knowledge Engineering. World Scientific Publishing Company. All, A., Castellar, E. P. N., & Looy, J. V. (2016). Assessing the effectiveness of digital game-based learning: Best practices. Computers & Education, 92–93, 90-103. Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman. Andrade, D. F., Tavares, H. R., Valle, R. C. (2000). Teoria de Resposta ao Item: conceitos e aplicações. ABE — Associação Brasileira de Estatística, 4o SINAPE. Andrade, H. & Valtcheva, A. (2009). Promoting learning and achievement through selfassessment. Theory into Practice, 48, 12-19. Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J. Marciniak, Encyclopedia of Software Engineering, (pp. 528-532). New York: Wiley-Interscience. Battistella, P. & Gresse von Wangenheim, C. (2016). Games for teaching computing in higher education – a systematic review. IEEE Technology and Engineering Education Journal, 9(1), 8-30. Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. Journal of Systems and Software, 76(3), 251-275. Bowman, D. D. (2018). Declining Talent in Computer Related Careers. Journal of Academic Administration in Higher Education, 14(1), 1-4. Boyle, E. A., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2, 69–74. Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 94, 178-192. Branch, R. M. (2010). Instructional Design: The ADDIE Approach. New York: Springer New York Dordrecht Heidelberg London. Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 47. Brown, G. T. L., & Harris, L. R. (2013). Student self-assessment. In J. H. McMillan (Ed.). The SAGE handbook of research on classroom assessment (pp. 367-393). Thousand Oaks: Sage Publications. Brown, G., Andrade, H., & Chen, F. (2015). Accuracy in student self-assessment: Directions and cautions for research. Assessment in Education Principles Policy and Practice, 22(4), 1-26. Budgen, D., Turner, M., Brereton, P., & Kitchenham, B. (2008). Using mapping studies in Software Engineering. Proc. of the 20th Annual Workshop of Psychology of Programming Interest Group, (pp. 195-204). Lancaster University, Lancaster, USA. Calderón, A. & Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396-422. Calderón, A., Ruiz M., & O'Connor, R. (2018). A multivocal literature review on serious games for software process standards education. Computer Standards & Interfaces, 57, 36-48. INCoD – Brazilian Institute for Digital Convergence JULY 2018 44 Çiftci, S. (2018). Trends of Serious Games Research from 2007 to 2017: A Bibliometric Analysis. Journal of Education and Training Studies, 6(2), 18-27. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319-340. Dawes, J. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50(1), 61-77. DeVellis, R. F. (2016). Scale development: theory and applications (4th. ed.). Thousand Oaks: SAGE Publications. Djaouti, D., Alvarez J., Jessel J. P., & Rampnoux O. (2011). Origins of Serious Games. In: Ma M., Oikonomou A., Jain L. (Eds) Serious Games and Edutainment Applications. London: Springer. Dolan, E. L. & Collins, J. P. (2015). We must teach more effectively: here are four ways to get st