The achievement of higher flexibility in multiple-choice-based tests using image classification techniques

In spite of the high accuracy of the existing optical mark reading (OMR) systems and devices, a few restrictions remain existent. In this work, we aim to reduce the restrictions of multiple-choice questions (MCQ) within tests. We use an image registration technique to extract the answer boxes from answer sheets. Unlike other systems that rely on simple image processing steps to recognize the extracted answer boxes, we address the problem from another perspective by training a machine learning classifier to recognize the class of each answer box (i.e., confirmed, crossed out or blank answer). This gives us the ability to deal with a variety of shading and mark patterns, and distinguish between chosen (i.e., confirmed) and canceled answers (i.e., crossed out). All existing machine learning techniques require a large number of examples in order to train a model for classification; therefore, we present a dataset including six real MCQ assessments with different answer sheet templates. We evaluate two strategies of classification: a straightforward approach and a two-stage classifier approach. We test two handcrafted feature methods and a convolutional neural network. At the end, we present an easy-to-use graphical user interface of the proposed system. Compared with existing OMR systems, the proposed system has the least constraints and achieves a high accuracy. We believe that the presented work will further direct the development of OMR systems toward reducing the restrictions of the MCQ tests.

[1]  Neil D. Lawrence,et al.  When Training and Test Sets Are Different: Characterizing Learning Transfer , 2009 .

[2]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[3]  Ming Liu,et al.  Automatic Chinese Multiple Choice Question Generation Using Mixed Similarity Strategy , 2018, IEEE Transactions on Learning Technologies.

[4]  Parinya Sanguansat.  Robust and low-cost Optical Mark Recognition for automated data entry , 2015, 2015 12th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON).

[5]  Abelardo Pardo,et al.  Grading Multiple Choice Exams with Low-Cost and Portable Computer-Vision Techniques , 2013 .

[6]  Daniel P. Lopresti,et al.  Towards Improved Paper-Based Election Technology , 2011, 2011 International Conference on Document Analysis and Recognition.

[7]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[8]  David G. Stork,et al.  Pattern Classification , 1973 .

[9]  Stephan Hussmann,et al.  A high-speed optical mark reader hardware implementation at low cost using programmable logic , 2005, Real Time Imaging.

[10]  Yosef A. Solewicz,et al.  Method of verifying declared identity in optical answer sheets , 2011, Soft Comput..

[11]  Daniel P. Lopresti,et al.  A Document Analysis System for Supporting Electronic Voting Research , 2008, 2008 The Eighth IAPR International Workshop on Document Analysis Systems.

[12]  Nobuyuki Otsu,et al.  ATlreshold Selection Method fromGray-Level Histograms , 1979 .

[13]  Amos Storkey,et al.  When Training and Test Sets are Different: Characterising Learning Transfer , 2013 .

[14]  Yuttapong Rangsanseri,et al.  Image-processing-oriented optical mark reader , 1999, Optics & Photonics.

[15]  Andrew M. Smith.  Optical mark reading - making it easy for users , 1981, SIGUCCS '81.

[16]  Tao Wang,et al.  End-to-end text recognition with convolutional neural networks , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[17]  Varin Chouvatut,et al.  The flexible and adaptive X-mark detection for the simple answer sheets , 2014, 2014 International Computer Science and Engineering Conference (ICSEC).

[18]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[19]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[20]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[21]  Hui Deng,et al.  A Low-Cost OMR Solution for Educational Applications , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing with Applications.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  P. McCoubrie.  Improving the fairness of multiple-choice questions: a literature review , 2004, Medical teacher.

[24]  Douglas Chai.  Automated marking of printed multiple choice answer sheets , 2016, 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE).

[25]  Norman Edward Gronlund,et al.  Assessment of student achievement , 1997 .

[26]  Daniel P. Lopresti,et al.  Mark detection from scanned ballots , 2009, Electronic Imaging.

[27]  Tien Dzung Nguyen,et al.  Efficient and reliable camera based multiple-choice test grading system , 2011, The 2011 International Conference on Advanced Technologies for Communications (ATC 2011).

[28]  Eugenio Culurciello,et al.  An Analysis of Deep Neural Network Models for Practical Applications , 2016, ArXiv.

[29]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[30]  Cordelia Schmid,et al.  An Affine Invariant Interest Point Detector , 2002, ECCV.

[31]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[32]  Mahmoud Afifi,et al.  OCR System for Poor Quality Images Using Chain-Code Representation , 2015, AISI.

[33]  Yasira Fathima,et al.  An Image Processing Oriented Optical Mark Reader , 2018 .

[34]  Andrea Spadaccini,et al.  A Multiple-Choice Test Recognition System based on the Gamera Framework , 2011, ArXiv.

[35]  G. Nagy,et al.  The Role of Document Image Analysis in Trustworthy Elections , 2014 .

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Francisco de Assis Zampirolli,et al.  An application for automatic multiple-choice test grading on android , 2016 .

[38]  Philip H. S. Torr,et al.  The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix , 1997, International Journal of Computer Vision.

[39]  Montse Maritxalar,et al.  Semantic Similarity Measures for the Generation of Science Tests in Basque , 2014, IEEE Transactions on Learning Technologies.

[40]  D. Lowe,et al.  Fast Matching of Binary Features , 2012, 2012 Ninth Conference on Computer and Robot Vision.

[41]  Andrew Zisserman,et al.  Reading Text in the Wild with Convolutional Neural Networks , 2014, International Journal of Computer Vision.

[42]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[43]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[44]  David D. Lewis,et al.  Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval , 1998, ECML.

[45]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.