Performance enhancement of the task assessment process through the application of an electronic performance support system

Higher education in Australia, as for many other countries, has changed greatly over the last 20 years at all levels and in many areas of operation including teaching, learning and assessment. The driving forces for these changes have been both internal and external, and have included factors such as: the increasing student population; the increasing use of part-time staff; a reduction in government funding; an increased expectation of institutional accountability; and the growing access and use of information and communication technologies (ICT) in teaching and learning. Assessment has not escaped these changes but in many cases has not kept up with exemplary and recommended practice. This is especially so in the specific area of task assessment that involves professional judgement, where assessment is a timeconsuming, expensive, poorly managed, and a stressful professional activity, and is often a negative emotional experience for both learners and educators. Professional judgement of educators in the task assessment process is becoming more important with the trend towards student-centred, standards-based curriculum and the use of authentic assessment tasks that are more subjective in nature. At the same time, stakeholders are demanding greater validity, reliability and transparency in the assessment process. To meet these demands, a new re-framing of the task assessment process is required, involving activities such as the design of the marking key, moderation, marking, feedback, reporting and management. Meanwhile, current methods and practices used in the task assessment process that involve professional judgement have not kept pace with current best practice, nor do they involve the application of ICT to any great extent. The study partly developed from the researcher’s professional reflection on the above issues. The study also developed from the researcher’s observation of the importance that electronic performance support systems (EPSS) and knowledge management (KM) have had in the commercial world in the area of improved performance of the worker and work processes. The premise of the study was to investigate to what extent the performance of the task assessment process involving professional judgement could be improved and enhanced through the application of an EPSS. A preliminary review of the literature identified three fields that needed to be reviewed, investigated and integrated for this study; these were: assessment of achievement, the use of EPSS, and software design and development.

[1]  Westone,et al.  Home Page , 2004, 2022 2nd International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA).

[2]  N. Bontis Managing organizational knowledge by diagnosing intellectual capital : framing and advancing the state of the field , 2000 .

[3]  Peter Knight,et al.  Summative Assessment in Higher Education: Practices in disarray , 2002 .

[4]  P. Taylor,et al.  Making Sense of Academic Life: Academics, Universities, and Change , 1999 .

[5]  Kate Ashcroft,et al.  Researching into assessment and evaluation in colleges and universities , 1996 .

[6]  Bonnie A. Nardi,et al.  Concepts of cognition and consciousness: four voices , 1998, ASTR.

[7]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[8]  P. Gates,et al.  The electronic, ‘paperless’ medical office; has it arrived? , 2007, Internal medicine journal.

[9]  Maria A. Jankowska,et al.  Identifying University Professors' Information Needs in the Challenging Environment of Information and Communication Technologies , 2004 .

[10]  Susan R. Furr,et al.  Suicide and depression among college students. , 1987 .

[11]  Janice Redish,et al.  User and task analysis for interface design , 1998 .

[12]  Albert N. Badre Shaping web usability---interaction design in context , 2002, INTR.

[13]  Darlene M. Van Tiem Interventions (Solutions) Usage and Expertise in Performance Technology Practice: An Empirical Investigation , 2008 .

[14]  Valentina Klenowski,et al.  Creating Communities of Shared Practice: The challenges of assessment use in learning and teaching , 2002 .

[15]  John B. Biggs,et al.  Teaching for Quality Learning at University: What the Student Does , 1999 .

[16]  Elizabeth Whitmore,et al.  Understanding and Practicing Participatory Evaluation. , 1998 .

[17]  Beverly Falk,et al.  Sitting Down To Score: Teacher Learning Through Assessment. , 1998 .

[18]  P. Knight,et al.  Quality and assessment , 1996 .

[19]  P. Knight The Value of a Programme-wide Approach to Assessment , 2000 .

[20]  K. Hambridge Action research. , 2000, Professional nurse.

[21]  Jon A. Preston,et al.  Improving on-line assessment: an investigation of existing marking methodologies , 1999, ITiCSE '99.

[22]  Thomas M. Schwen,et al.  Potential knowledge management contributions to human performance technology research and practice , 1998 .

[23]  Joanna Bull,et al.  Computer-assisted assessment in higher education , 1999 .

[24]  Jeanne Farrington,et al.  From Training to Performance Improvement: Navigating the Transition , 2007 .

[25]  Barry Raybould,et al.  Performance Support Engineering: Building Performance-Centered Web-based Systems, Information Systems, and Knowledge Management Systems in the 21st Century. , 2000 .

[26]  Claire Simpson,et al.  Measuring the response of students to assessment: the Assessment Experience Questionnaire , 2003 .

[27]  Jon H Sims Williams,et al.  Computer-Assisted Assessment in Higher Education , 1999 .

[28]  Thomas C. Reeves,et al.  A Research Agenda for Interactive Learning in the New Millennium , 1999 .

[29]  William J. Rothwell Beyond Training and Development: The Groundbreaking Classic on Human Performance Enhancement , 2005 .

[30]  Abigail Sellen,et al.  The myth of the paperless office mit press , 2003 .

[31]  J. R. Christie,et al.  Automated Essay Marking for Content ~ does it work? , 2003 .

[32]  Sandra McCollister,et al.  Developing Criteria Rubrics in the Art Classroom , 2002 .

[33]  L. Cohen,et al.  Research Methods in Education , 1980 .

[34]  Penny Baillie de Byl An Online Assistant for Remote, Distributed Critiquing of Electronically Submitted Assessment , 2004, J. Educ. Technol. Soc..

[35]  Jonathan Stars,et al.  Learn FileMaker Pro 7 , 2002 .

[36]  L. Hayward,et al.  Travelling towards change in assessment: Policy, practice and research in education , 2005 .

[37]  Harold D. Stolovitch,et al.  Handbook of human performance technology: Improving individual and organizational performance worldwide (second edition) , 1999 .

[38]  Stephen J. Sharp Sally Brown and Angela Glasner (eds.) 1999. Assessment Matters in Higher Education -- Choosing and Using Diverse Approaches , 1999 .

[39]  Rosa Maria Bottino,et al.  The evolution of ICT-based learning environments: which perspectives for the school of the future? , 2004, Br. J. Educ. Technol..

[40]  James L. Moseley,et al.  Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance , 2000 .

[41]  H. Tillema Collaborative Knowledge Construction in Study Teams of Professionals , 2005 .

[42]  Alison A. Carr,et al.  User-design in the creation of human learning systems , 1997 .

[43]  G. Wiggins Assessing student performance : exploring the purpose and limits of testing , 1993 .

[44]  Janice Orrell,et al.  Feedback on learning achievement: rhetoric and reality , 2006 .

[45]  Gary J. Anderson,et al.  Fundamentals of Educational Research , 1998 .

[46]  Bruce W. Speck,et al.  Learning‐teaching‐assessment paradigms and the on‐line classroom , 2002 .

[47]  A. McFarlane Perspectives on the relationships between ICT and assessment , 2001, J. Comput. Assist. Learn..

[48]  Anne P. Massey,et al.  Human Performance Technology and Knowledge Management: A Case Study , 2008 .

[49]  서정목,et al.  문화적 차이에 따른 Business Communication , 1996 .

[50]  C. Palau,et al.  ASSESSMENT IN EDUCATION , 2015 .

[51]  K. Frame,et al.  Making the Grade‐‐Developing Grade Descriptors for Accounting using a Discriminator Model of Performance , 2001 .

[52]  A. Cattell The Adult Learner – The Definitive Classic in Adult Education and Human Resource Development , 2005 .

[53]  Gloria J. Gery,et al.  Electronic performance support systems: how and why to remake the workplace through the strategic application of technology , 1991 .

[54]  Peter Knight,et al.  Assessment for learning in higher education , 1998 .

[55]  Mark N. K. Saunders,et al.  THE USE OF ASSESSMENT CRITERIA TO ENSURE CONSISTENCY OF MARKING , 1998 .

[56]  Sally Brown,et al.  Assessment Matters in Higher Education: Choosing and Using Diverse Approaches. , 1999 .

[57]  Alexandre B. Lopes,et al.  The Knowing-Doing Gap: How Smart Companies Turn Knowledge into Action , 1999 .

[58]  Kevin Cox,et al.  Student Assessment in Higher Education: A Handbook for Assessing Performance , 1998 .

[59]  Harold D. Stolovitch,et al.  Training Ain't Performance , 2006 .

[60]  David Kember,et al.  To Control or Not to Control: The question of whether experimental designs are appropriate for evaluating teaching innovations in higher education , 2003 .

[61]  N. Staggers INTERNET-BASED ORGANIZATIONAL MEMORY AND KNOWLEDGE MANAGEMENT , 2002 .

[62]  P. Black Research and the Development of Educational Assessment , 2000 .

[63]  J. Lumby,et al.  Practitioner Research in Education: Making a Difference , 1999 .

[64]  Thomas F. Gilbert,et al.  Human Competence: Engineering Worthy Performance , 1978 .

[65]  Mantz Yorke,et al.  Discipline‐related Marking Behaviour Using Percentages: a potential cause of inequity in assessment , 1999 .

[66]  Steve Armstrong,et al.  Facing up to radical changes in universities and colleges , 1997 .

[67]  Pete Cannell,et al.  Practitioner research and professional development in education , 2004, Br. J. Educ. Technol..

[68]  Alistair Campbell,et al.  Application of ICT and rubrics to the assessment process where professional judgement is involved: the features of an e‐marking tool , 2005 .

[69]  Bruce Saddler,et al.  The Writing Rubric. , 2004 .

[70]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[71]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[72]  G. Gibbs,et al.  Conditions Under Which Assessment Supports Students’ Learning , 2005 .

[73]  J. Herman,et al.  What Does Research Say About Assessment , 2003 .

[74]  James L. Moseley,et al.  Performance Improvement Interventions: Enhancing People, Processes, and Organizations through Performance Technology , 2006 .

[75]  C. Gipps Beyond Testing: Towards a Theory of Educational Assessment , 1994 .

[76]  Donald A. Norman,et al.  Things that make us smart , 1979 .

[77]  J. H. McMillan,et al.  Understanding and Evaluating Educational Research , 1998 .

[78]  Denise M. Woit,et al.  Effectiveness of online assessment , 2003, SIGCSE.

[79]  Alistair Campbell Augmentation of the Assessment Process by the use of an Electronic Performance Support System , 2003 .

[80]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[81]  Jim Ridgway,et al.  Literature review of e-assessment. , 2004 .

[82]  Alistair Campbell Task assessment at your fingertips , 2005 .

[83]  David Boud,et al.  Enhancing learning through self assessment , 1995 .

[84]  Gloria Gery,et al.  Performance support in internet time the state of the practice , 2000 .

[85]  T.L.J. Ferris,et al.  User-Centered Design: An Integrated Approach , 2004, IEEE Transactions on Professional Communication.

[86]  Kevin Bampton A low technology computer assisted marking strategy for law essays , 2004 .

[87]  Susan R. Furr,et al.  Suicide and depression among college students: A decade later. , 2001 .

[88]  Richard Freeman,et al.  Planning and Implementing Assessment , 1998 .

[89]  Deborah G. Coffin Saving Time with a Rubric Buffet , 2002 .

[90]  Patricia Broadfoot,et al.  Education, assessment, and society : a sociological analysis , 1996 .

[91]  Harold D. Stolovitch,et al.  Human performance technology: Research and theory to practice , 2000 .

[92]  Robert H. Ennis,et al.  Philosophy of educational research , 1973 .

[93]  Frank Webster,et al.  Assessing the Undergraduate Dissertation , 2000 .

[94]  R. Jacobs What to research , 2009 .

[95]  Kathryn Ecclestone,et al.  'I know a 2:1 when I see it': Understanding criteria for degree classifications in franchised university programmes , 2001 .

[96]  T. Urdan,et al.  Corporate elearning: exploring a new frontier , 2000 .

[97]  William Wiersma,et al.  Research Methods in Education: An Introduction , 1980 .

[98]  James R. Rudd,et al.  The Art of Rapid Prototyping: User Interface Design for Windows and Os/2 , 1996 .

[99]  Mantz Yorke,et al.  The Management of Assessment in Higher Education , 1998 .

[100]  K. Howe Closing Methodological Divides: Toward Democratic Educational Research , 2003 .

[101]  David Birchall,et al.  Transfer of Learning from Management Development Programmes: Testing the Holton Model , 2006 .

[102]  Robert Cannon,et al.  A Handbook for Teachers in Universities & Colleges: A Guide To Improving Teaching Methods. Third Edition. , 1992 .

[103]  Marcia Devlin,et al.  Assessing learning in Australian universities , 2002 .

[104]  Wilfred Carr,et al.  For Education: Towards Critical Educational Inquiry , 1995 .

[105]  Gretchen B. Rossman,et al.  Designing qualitative research, 3rd ed. , 1999 .

[106]  Kent L. Gustafson Designing technology-based performance support , 2000 .

[107]  R. Kuisma,et al.  Criteria Referenced Marking of Written Assignments , 1999 .

[108]  M. Brennan Book Review: A Handbook for Teaching and Learning in Higher Education. Enhancing Academic Practice , 2002, Journal of orthodontics.

[109]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[110]  D. Boud Assessment and learning: contradictory or complementary? , 1995 .

[111]  Heidi Andrade,et al.  Role of Rubric-Referenced Self-Assessment in Learning to Write , 2003 .

[112]  Shelley Paewai,et al.  Academic Staff Workloads and Job Satisfaction: Expectations and values in academe , 2006 .

[113]  P. Black Formative and Summative Assessment by Teachers , 1993 .

[114]  Grant Wiggins,et al.  Educative Assessment: Designing Assessments to Inform and Improve Student Performance , 1998 .

[115]  David Wile Why Doers Do. , 1996 .

[116]  Richard Winter,et al.  Contextualizing the Patchwork Text: addressing problems of coursework assessment in higher education , 2003 .

[117]  D. Carless Differing perceptions in the feedback process , 2006 .

[118]  Chris Rust,et al.  Improving Students' Learning by Developing their Understanding of Assessment Criteria and Processes , 2003 .

[119]  Mantz Yorke Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice , 2003 .

[120]  Christine Reid,et al.  The Myth of the Paperless Office , 2003, J. Documentation.

[121]  What Works with Authentic Assessment , 2003 .

[122]  Kathleen Montgomery,et al.  Authentic Tasks and Rubrics: Going Beyond Traditional Assessments in College Teaching , 2002 .

[123]  L. Murray,et al.  Practitioner-Based Enquiry: Principles for Postgraduate Research , 2000 .

[124]  Adam Warren,et al.  Integr@ting technology in learning & teaching. A practical guide for educators , 2000 .

[125]  P. Knight,et al.  Assessing learners in higher education , 1998 .

[126]  Joanna Bull,et al.  Assessing student learning in higher education , 1997 .

[127]  M. Simon,et al.  What's still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels , 2004 .

[128]  Susan A. Santo Knowledge management: An imperative for schools of education , 2005 .

[129]  Mary L. Broad Beyond Transfer of Training: Engaging Systems to Improve Performance , 2005 .

[130]  Joi L. Moore,et al.  The development of an electronic performance support system for teachers , 2000 .

[131]  Jarkko Suhonen,et al.  Using Technology in Teaching , 2006, J. Educ. Technol. Soc..

[132]  Wim Jochems,et al.  Integrated E-Learning: Implications for Pedagogy, Technology and Organization , 2004 .

[133]  Caroline V. Gipps,et al.  What is the role for ICT-based assessment in universities? , 2005 .

[134]  Nigel Bevan,et al.  International standards for HCI and usability , 2001, Int. J. Hum. Comput. Stud..

[135]  Norman G. Lederman,et al.  Examining Pedagogical Content Knowledge: The Construct and its Implications for Science Education , 2001 .

[136]  Cathy Nutbrown,et al.  A Student's Guide to Methodology , 2002 .

[137]  Katherine Samuelowicz,et al.  Identifying academics' orientations to assessment practice , 2002 .

[138]  J. Galbreath Knowledge management technology in education: An overview , 2000 .

[139]  J. C. R. Licklider,et al.  Man-Computer Symbiosis , 1960 .

[140]  Gary J. Dickelman,et al.  Distributed Cognition: A Foundation for Performance Support , 2000 .

[141]  F Vella,et al.  A handbook for teachers in universities and colleges: A guide to improving teaching methods: By D Newble and R Cannon. pp 159. St Martin's Press, New York. 1989 ISBN 0‐312‐03196‐3 , 1990 .

[142]  John Elliott,et al.  Action research for educational change , 1991 .

[143]  John A Bowden,et al.  Action Research: A Pathway to Action, Knowledge and Learning , 1999 .

[144]  H. Andrade Teaching With Rubrics: The Good, the Bad, and the Ugly , 2005 .

[145]  C Winslow,et al.  Futurework : Putting Knowledge To Work In the Knowledge Economy , 1994 .

[146]  Chris Rust,et al.  Know what I mean? Enhancing student understanding of assessment standards and criteria , 2004 .

[147]  B. Hughes Study, examinations, and stress: Blood pressure assessments in college students , 2005 .