A Survey on Online Judge Systems and Their Applications

Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.

[1]  Niklaus Wirth,et al.  Automatic grading programs , 1965, CACM.

[2]  Jacek Blazewicz,et al.  Computer Representations of Bioinformatics Models , 2016 .

[3]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[4]  Chris Wilcox.  Testing Strategies for the Automated Grading of Student Programs , 2016, SIGCSE.

[5]  José Paulo Leal,et al.  Mooshak: a Web‐based multi‐site programming contest system , 2003, Softw. Pract. Exp..

[6]  Guojin Zhu,et al.  Automatic Organization of Programming Resources on the Web , 2012 .

[7]  Jacek Blazewicz,et al.  E-Commerce Evaluation - Multi-Item Internet Shopping. Optimization and Heuristic Algorithms , 2010, OR.

[8]  Thomas A. Runkler,et al.  Rescheduling and optimization of logistic processes using GA and ACO , 2008, Eng. Appl. Artif. Intell..

[9]  Salvador Roura,et al.  Better Feedback for Educational Online Judges , 2014, CSEDU.

[10]  Edmund K. Burke,et al.  A Research Agenda for Metaheuristic Standardization , 2015 .

[11]  A. Kaplan,et al.  Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the Cookie Monster , 2016 .

[12]  Yingwei Luo,et al.  Programming grid: a computer-aided education system for programming courses based on online judge , 2008, SCE '08.

[13]  M. Hagberg.  Editorial , 2004 .

[14]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[15]  Developing Predictive Molecular Maps of Human Disease through Community-based Modeling , 2011 .

[16]  urs von Matt.  Kassandra: the automatic grading system , 1994 .

[17]  Andrew Lim,et al.  Online Judge , 2001, Comput. Educ..

[18]  Yongqiang Chen,et al.  Online Learning Management System , 2015, 2015 International Conference on Computational Science and Computational Intelligence (CSCI).

[19]  Jacek Blazewicz,et al.  Novel dual discounting functions for the Internet shopping optimization problem: new algorithms , 2014, Journal of Scheduling.

[20]  Vivek Khera,et al.  The internet programming contest: a report and philosophy , 1993, SIGCSE '93.

[21]  Carles Creus López,et al.  Automatic evaluation of context-free grammars (system description) , 2014 .

[22]  Stephen H. Edwards,et al.  Web-CAT: automatically grading programming assignments , 2008, ITiCSE.

[23]  Andrew Lim,et al.  On automated grading of programming assignments in an academic institution , 2003, Comput. Educ..

[24]  J. Edmonds.  Paths, Trees, and Flowers , 1965, Canadian Journal of Mathematics.

[25]  Jürgen Börstler,et al.  Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies , 2015, ITiCSE-WGR.

[26]  Shahida Sulaiman,et al.  Automatic programming assessment and test data generation a review on its approaches , 2010, 2010 International Symposium on Information Technology.

[27]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[28]  Jacek Blazewicz,et al.  RNAssess—a web server for quality assessment of RNA 3D structures , 2015, Nucleic Acids Res..

[29]  Luonan Chen,et al.  Optimization meets systems biology , 2010, BMC Systems Biology.

[30]  Zenón José Hernández Figueroa,et al.  A Virtual Programming Lab for Moodle with automatic assessment and anti-plagiarism features , 2012 .

[31]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[32]  Jimmy J. Lin,et al.  Report on the Evaluation-as-a-Service (EaaS) Expert Workshop , 2015, SIGIR Forum.

[33]  Maciej Drozdowski,et al.  Analysis and Solution of CSS-Sprite Packing Problem , 2015, TWEB.

[34]  Jacek Blazewicz,et al.  Hypercycle , 2016, PLoS Comput. Biol..

[35]  Chao Yi,et al.  A Comparison of Sandbox Technologies Used in Online Judge Systems , 2014 .

[36]  J C Costello,et al.  Seeking the Wisdom of Crowds Through Challenge‐Based Competitions in Biomedical Research , 2013, Clinical pharmacology and therapeutics.

[37]  Luís Torgo,et al.  OpenML: A Collaborative Science Platform , 2013, ECML/PKDD.

[38]  Lennart E. Nacke,et al.  From game design elements to gamefulness: defining "gamification" , 2011, MindTrek.

[39]  Thomas Vikberg,et al.  Test My Code , 2013 .

[40]  Adrian Kosowski,et al.  Application of an Online Judge & Contester System in Academic Tuition , 2007, ICWL.

[41]  Steve Benford,et al.  Learning to construct quality software with the Ceilidh system , 1993, Software Quality Journal.

[42]  Anthony Goldbloom,et al.  Data Prediction Competitions -- Far More than Just a Bit of Fun , 2010, 2010 IEEE International Conference on Data Mining Workshops.

[43]  Mümine Kaya,et al.  Integrating an online compiler and a plagiarism detection tool into the Moodle distance education system for easy assessment of programming assignments , 2015, Comput. Appl. Eng. Educ..

[44]  Ramakrishnan Rajamony,et al.  An updated performance comparison of virtual machines and Linux containers , 2015, 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[45]  Eric Bourreau,et al.  Disruption management for commercial airlines: methods and results for the ROADEF 2009 Challenge , 2012 .

[46]  Szymon Wasik,et al.  Optil.io: Cloud Based Platform For Solving Optimization Problems Using Crowdsourcing Approach , 2016, CSCW '16 Companion.

[47]  Wolfgang Pohl,et al.  Computer Science Contests for Secondary School Students: Approaches to Classification , 2006, Informatics Educ..

[48]  Christoph Meinel,et al.  Towards practical programming exercises and automated assessment in Massive Open Online Courses , 2015, 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE).

[49]  A. Elo.  The rating of chessplayers, past and present , 1978 .

[50]  Su Xiaohong,et al.  Moodle Plugins for Highly Efficient Programming Courses , 2001 .

[51]  Teemu Koskinen,et al.  Service-Oriented Approach to Improve Interoperability of E-Learning Systems , 2013, 2013 IEEE 13th International Conference on Advanced Learning Technologies.

[52]  Ning Guo,et al.  Enhance performance of program automatic online judging systems using affinity algorithm and queuing theory in SMP environment , 2011, Proceedings of 2011 International Conference on Electronic & Mechanical Engineering and Information Technology.

[53]  Alda Lopes Gançarski,et al.  A Flexible Dynamic System for Automatic Grading of Programming Exercises , 2013, SLATE.

[54]  Michael Luck,et al.  On-line submission and testing of programming assignments , 1995 .

[55]  Prathibha Davuluri,et al.  Moodle Java Autograder , 2016 .

[56]  Sebastian Stawicki,et al.  TunedIT.org: System for Automated Evaluation of Algorithms in Repeatable Experiments , 2010, RSCTC.

[57]  Scott Sanner,et al.  A Survey of the Seventh International Planning Competition , 2012, AI Mag..

[58]  C. Lokhorst,et al.  Knowledge Discovery in Dutch Dairy Databases , 1998 .

[59]  Jacek Blazewicz,et al.  ModeLang: A New Approach for Experts-Friendly Viral Infections Modeling , 2013, Comput. Math. Methods Medicine.

[60]  Gary Geunbae Lee,et al.  An effective procedure for constructing a hierarchical text classification system , 2006 .

[61]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[62]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[63]  GannonDennis,et al.  Programming the Grid , 2002 .

[64]  Guo Wei,et al.  Peking University Oneline Judge and Its Applications , 2005 .

[65]  Rita Kop,et al.  The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course , 2011 .

[66]  Charlie Daly,et al.  RoboProf and an introductory computer programming course , 1999, ITiCSE '99.

[67]  Salvador Roura,et al.  Jutge.org: an educational programming judge , 2012, SIGCSE '12.

[68]  Gordon V. Cormack,et al.  Structure, Scoring and Purpose of Computing Competitions , 2006, Informatics Educ..

[69]  Michal Forǐsek,et al.  Security of Programming Contest Systems , 2007 .

[70]  J. Xavier,et al.  COMPUTER-BASED ASSESSMENT SYSTEM FOR E-LEARNING APPLIED TO PROGRAMMING EDUCATION , 2011 .

[71]  Jacek Blazewicz,et al.  Structural alignment of protein descriptors – a combinatorial model , 2016, BMC Bioinformatics.

[72]  Julio C. Caiza,et al.  PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS , 2013 .

[73]  O. Erhun Kundakcioglu,et al.  Combinatorial Optimization in Data Mining , 2013 .

[74]  Gottfried Vossen,et al.  XLX-a platform for graduate-level exercises , 2002, International Conference on Computers in Education, 2002. Proceedings..

[75]  Manuel Moreno Carral.  Moodle autograder plugin , 2013 .

[76]  Mümine Kaya,et al.  An online compiler module for grading programming assignments on Moodle distance education system , 2012 .

[77]  Yu Jiang,et al.  A code reviewer assignment model incorporating the competence differences and participant preferences , 2016 .

[78]  Bertrand Meyer,et al.  An Incremental Hint System For Automated Programming Assignments , 2015, ITiCSE.

[79]  Daniel Shawcross Wilkerson,et al.  Winnowing: local algorithms for document fingerprinting , 2003, SIGMOD '03.

[80]  Inggriani Liem,et al.  Scalable Autograder and LMS Integration , 2013 .

[81]  B. P. S. Sahoo,et al.  Cloud Computing Features, Issues, and Challenges: A Big Picture , 2015, 2015 International Conference on Computational Intelligence and Networks.

[82]  Szymon Wasik,et al.  Inferring Mathematical Equations Using Crowdsourcing , 2015, PloS one.

[83]  Vreda Pieterse,et al.  Automated Assessment of Programming Assignments , 2013, CSERC.

[84]  Thomas Ilsche,et al.  Combining Instrumentation and Sampling for Trace-Based Application Performance Analysis , 2015 .

[85]  Nimrod Megiddo,et al.  On the complexity of locating linear facilities in the plane , 1982, Oper. Res. Lett..

[86]  Vasant Dhar,et al.  Data science and prediction , 2012, CACM.

[87]  Craig A. Knoblock,et al.  PDDL-the planning domain definition language , 1998 .

[88]  Allan Hanbury,et al.  Report on the Cloud-Based Evaluation Approaches Workshop 2015 , 2016, SIGIR Forum.

[89]  F. Bell.  Connectivism: Its place in theory-informed research and innovation in technology-enabled learning , 2011 .

[90]  Ágnes Erdősné Németh,et al.  Online Training and Contests for Informatics Contestants of Secondary School Age , 2015 .

[91]  Michal Forisek,et al.  On the Suitability of Programming Tasks for Automated Evaluation , 2006, Informatics Educ..

[92]  Sébastien Combéfis,et al.  Programming Trainings and Informatics Teaching Through Online Contests , 2014 .

[93]  Araz Taeihagh,et al.  MOOCs and Crowdsourcing: Massive Courses and Massive Resources , 2015, First Monday.

[94]  Steven Skiena,et al.  Programming challenges: the programming contest training manual , 2003, SIGA.

[95]  Nathan Griffiths,et al.  The boss online submission and assessment system , 2005, JERC.

[96]  S. Friend,et al.  Crowdsourcing biomedical research: leveraging communities as innovation engines , 2016, Nature Reviews Genetics.

[97]  Hamid R. Arabnia,et al.  E-Learning, E-Business, Enterprise Information Systems, and E-Government , 2015 .

[98]  David Hovemeyer,et al.  Analyzing Student Work Patterns Using Programming Exercise Data , 2015, SIGCSE.

[99]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[100]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..