Construction and analysis of educational assessments using knowledge maps with weight appraisal of concepts

The rapid advance of information and communication technologies (ICT) has important impacts on teaching and learning, as well as on the educational assessment. Teachers may create assessments utilizing some developed assessment software or test authoring tools. However, problems could occur, such as neglecting key concepts in the curriculum or having disproportionate course topics distribution, when teachers create assessments or test items. This study proposes a novel approach, which uses knowledge map with appraisal of concept weights and other ICTs, and implements an assessment system KMAAS to help primary school teachers in Taiwan, or elsewhere, create educational assessments properly. When compiling an assessment, KMAAS ensures that teachers can include all important course concepts intended for assessing and maintain correct balance between course concepts among test items. It does so first by analyzing course material of the assessment range and displaying a concept-weight-annotated knowledge map which concretize and visualize the importance of and the relationships among concepts in the range. It then analyzes the test sheet which is being complied and displays another similar real-time updated knowledge map containing balance between course concepts among the test items. Teachers may cross-refer to these maps to help them adjust concept balances and even select appropriate test items from test banks. The system has being evaluated in both the accuracy of learning concepts extraction and the degree of user satisfaction, as measured by questionnaires given to the teachers who tested the system. The promising results confirm the feasibility of this system in helping teachers compile their educational assessments easily and precisely. Other results of the formative evaluations on techniques have being used to improve the system in order to make it more effective and efficient. The methodology and technologies KMAAS employed are all well developed and are domain independent, which makes it highly flexible to transfer to other course subject domain too.

[1]  Julia Phelps,et al.  Formative evaluation of a Web-based course in meteorology , 1999, Comput. Educ..

[2]  Gwo-Jen Hwang,et al.  An innovative parallel test sheet composition approach to meet multiple assessment criteria for national tests , 2008, Comput. Educ..

[3]  Donald F. Dansereau,et al.  Multirelational semantic maps , 1989 .

[4]  Maiga Chang,et al.  Analyzing Problem's Difficulty based on Neural Networks and Knowledge Map , 2004, J. Educ. Technol. Soc..

[5]  Mark T. Maybury,et al.  Advances in Automatic Text Summarization , 1999 .

[6]  El-Sayed M. El-Alfy,et al.  Construction and analysis of educational tests using abductive machine learning , 2008, Comput. Educ..

[7]  Chun-Yen Chang,et al.  Assessing Creative Problem-solving with Automated Text Grading , 2008 .

[8]  Gwo-Jen Hwang,et al.  Diagnosing student learning problems based on historical assessment records , 2008 .

[9]  Dirk Ifenthaler,et al.  Automated Knowledge Visualization and Assessment , 2010 .

[10]  Nian-Shing Chen,et al.  Mining e-Learning Domain Concept Map from Academic Articles , 2006 .

[11]  Damras Wongsawang,et al.  Adaptable learning assistant for item bank management , 2008, Comput. Educ..

[12]  Stephen H. Edwards,et al.  Designing an adaptive learning module to teach software testing , 2006, SIGCSE '06.

[13]  Joseph D. Novak,et al.  Position papers: The reception learning paradigm , 1979 .

[14]  Julie H. Lester.  Planning Effective Secondary Professional Development Programs , 2003 .

[15]  D. Dansereau,et al.  A Convergent Paradigm for Examining Knowledge Mapping as a Learning Strategy , 1991 .

[16]  Sylvie Ranwez,et al.  Concept Maps for Designing Adaptive Knowledge Maps , 2006, Inf. Vis..

[17]  Jörg Zumbach,et al.  The role of graphical and text based argumentation tools in hypermedia learning , 2009, Comput. Hum. Behav..

[18]  Euiho Suh,et al.  Building the knowledge map: an industrial case study , 2003, J. Knowl. Manag..

[19]  Suzanne Rivard,et al.  A Keyword Classification Scheme for IS Research Literature: An Update , 1993 .

[20]  Sheizaf Rafaeli,et al.  On-line question-posing and peer-assessment as means for web-based knowledge sharing in learning , 2004, Int. J. Hum. Comput. Stud..

[21]  Keh-Jiann Chen,et al.  Unknown Word Extraction for Chinese Documents , 2002, COLING.

[22]  Lydia M. Conca *,et al.  Challenges teachers face as they work to connect assessment and instruction , 2004 .

[23]  Gwo-Jen Hwang,et al.  A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria , 2006, J. Educ. Technol. Soc..

[24]  Angus S. McDonald,et al.  The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments , 2002, Comput. Educ..

[25]  Robert J. Beichner,et al.  Education Research Using Web-Based Assessment Systems , 2000 .

[26]  Yen-Ting Lin,et al.  An adaptive testing system for supporting versatile educational assessment , 2009, Comput. Educ..

[27]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[28]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[29]  Chun-Hua Chen,et al.  Developing an Intelligent Diagnosis and Assessment E-learning Tool for Introductory Programming , 2008, J. Educ. Technol. Soc..

[30]  Qingping He,et al.  A computer-assisted test design and diagnosis system for use by classroom teachers , 2005, J. Comput. Assist. Learn..

[31]  W. James Popham.  The Age of Compliance. , 2005 .

[32]  Richard H. Hall,et al.  A Navigational Analysis of Linear and Non-Linear Hypermedia Interfaces , 2000 .

[33]  José Chamoso,et al.  Analysis of the reflections of student-teachers of mathematics when working with learning portfolios in Spanish university classrooms , 2009 .

[34]  Al Saif,et al.  A Proposal for a Preventive Program for Risks Associated with the Use of the Internet and Its Impact upon Students' Awareness of Perverse Issues. , 2010 .

[35]  Jia-Sheng Heh,et al.  Implementing a Problem Solving System for Physics based on Knowledge Map and Four Steps Problem Solving Strategies , 2004 .

[36]  Thomas H. Davenport,et al.  Book review:Working knowledge: How organizations manage what they know. Thomas H. Davenport and Laurence Prusak. Harvard Business School Press, 1998. $29.95US. ISBN 0‐87584‐655‐6 , 1998 .

[37]  Jennifer K Knight,et al.  The Genetics Concept Assessment: a new concept inventory for gauging student understanding of genetics. , 2008, CBE life sciences education.

[38]  Lynda Hardman,et al.  'Lost in Hyperspace': Cognitive Mapping and Navigation in a Hypertext Environment , 1999, UK Hypertext.

[39]  Craig A. Mertler.  Secondary Teachers' Assessment Literacy: Does Classroom Experience Make A Difference? , 2004 .

[40]  Chien-Hung Liu,et al.  Learning effectiveness in a Web-based virtual learning environment: a learner control perspective , 2005, J. Comput. Assist. Learn..

[41]  G. Conole,et al.  A review of computer-assisted assessment , 2005 .

[42]  Joseph D. Novak,et al.  Learning How to Learn , 1984 .

[43]  H. Alshawi,et al.  The Core Language Engine , 1994 .

[44]  Fred D. Davis,et al.  Extension of the Technology Acceptance Model: Four Longitudinal Field : . , 2000 .

[45]  Keh-Jiann Chen,et al.  Introduction to CKIP Chinese Word Segmentation System for the First International Chinese Word Segmentation Bakeoff , 2003, SIGHAN.

[46]  Gráinne Conole,et al.  A review of computer-assisted assessment , 2005 .

[47]  Nigel Shadbolt,et al.  Knowledge Mapping for Industrial Purposes , 1999 .

[48]  Ruey-Shiang Shaw,et al.  A study of learning performance of e-learning materials design with knowledge maps , 2010, Comput. Educ..

[49]  James W. Popham,et al.  Needed: A Dose of Assessment Literacy. , 2006 .

[50]  Cheri A. Hoy,et al.  Graphic Organizers Applied to Higher–Level Secondary Mathematics , 2003 .

[51]  Angela M. O'Donnell,et al.  Knowledge Maps as Scaffolds for Cognitive Processing , 2002 .

[52]  Roberto Basili,et al.  Lexical Acquisition and Information Extraction , 1997, SCIE.

[53]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .

[54]  Kevin Knight,et al.  Mining online text , 1999, Commun. ACM.

[55]  Hans Peter Luhn,et al.  The Automatic Creation of Literature Abstracts , 1958, IBM J. Res. Dev..

[56]  Kenneth R. Boff,et al.  Knowledge maps for knowledge mining: application to R&D/technology management , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[57]  Tianyong Hao,et al.  Automatic categorization of questions for user-interactive question answering , 2011, Inf. Process. Manag..

[58]  Dan Roth,et al.  Learning Question Classifiers , 2002, COLING.

[59]  Joseph D. Novak,et al.  The Reception Learning Paradigm. , 1979 .

[60]  Dell Zhang,et al.  Question classification using support vector machines , 2003, SIGIR.

[61]  William R. King,et al.  A meta-analysis of the technology acceptance model , 2006, Inf. Manag..

[62]  Timothy Teo,et al.  Assessing the intention to use technology among pre-service teachers in Singapore and Malaysia: A multigroup invariance analysis of the Technology Acceptance Model (TAM) , 2009, Comput. Educ..

[63]  Mervat Gheith,et al.  Lexical Acquisition for Information Extraction from Arabic Text Documents , 2004 .

[64]  T. A. Goodison,et al.  Learning with ICT at primary level: pupils' perceptions , 2002, J. Comput. Assist. Learn..

[65]  Dirk Ifenthaler,et al.  Computer-Based Diagnostics and Systematic Analysis of Knowledge , 2010 .