Achieving improved quality and validity: reframing research and evaluation of learning technologies.

A critical reading of research literature relating to teaching and learning with technology for open, distance and blended education reveals a number of shortcomings in how investigations are conceptualised, conducted and reported. Projects often lack clarity about the nature of the enhancement that technology is intended to bring about. Frequently there is no explicit discussion of assumptions and beliefs that underpin research studies and the approaches used to investigate the educational impact of technologies. This presentation summarises a number of the weaknesses identified in published studies and considers the implications. Some ways in which these limitations could be avoided through a more rigorous approach to undertaking research and evaluation studies are then outlined and discussed.

[1]  Adrian Kirkwood,et al.  Assessment and student learning: a fundamental relationship and the role of information and communication technologies , 2008 .

[2]  K. Howe.  Positivist Dogmas, Rhetoric, and the Education Science Question , 2009 .

[3]  R. Slavin.  A Reader's Guide to Scientifically Based Research. , 2003 .

[4]  Carole Torgerson,et al.  The Need for Randomised Controlled Trials in Educational Research , 2001 .

[5]  M. Oliver,et al.  Technological determinism in educational technology research: some alternative ways of thinking about the relationship between learning and technology , 2011, J. Comput. Assist. Learn..

[6]  Linda Price,et al.  Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review , 2014 .

[7]  Joanna Bull,et al.  Assessing student learning in higher education , 1997 .

[8]  RadhaKanta Mahapatra,et al.  EVALUATING TRAINING PROGRAMS , 2005 .

[9]  M. Scriven.  A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research , 2008, Journal of MultiDisciplinary Evaluation.

[10]  T. Reeves.  Can Educational Research Be Both Rigorous and Relevant , 2011 .

[11]  Philip C. Abrami,et al.  What Forty Years of Research Says About the Impact of Technology on Learning , 2011 .

[12]  Anton Havnes,et al.  Examination and learning: an activity‐theoretical analysis of the relationship between assessment and educational practice , 2004 .

[13]  G. Biesta.  WHY “WHAT WORKS” WON’T WORK: EVIDENCE‐BASED PRACTICE AND THE DEMOCRATIC DEFICIT IN EDUCATIONAL RESEARCH , 2007 .

[14]  H. Simons.  Evidence‐based practice: panacea or over promise? , 2003 .

[15]  Adrian Kirkwood,et al.  Examining some assumptions and limitations of research on the effects of emerging technologies for teaching and learning in higher education , 2013, Br. J. Educ. Technol..

[16]  Liz McDowell,et al.  The construction of the hidden curriculum: messages and meanings in the assessment of student learning , 1998 .

[17]  Margaret J. Cox,et al.  Effects of ICT: Do we know what we should know? , 2007, Education and Information Technologies.

[18]  S. Clegg.  Evidence‐based practice in educational research: a critical realist critique of systematic review , 2005 .

[19]  N. Selwyn.  Ten suggestions for improving academic research in education and technology , 2012 .

[20]  Adrian Kirkwood,et al.  Using technology for teaching and learning in higher education: a critical review of the role of evidence in informing practice , 2014 .

[21]  K. Trigwell,et al.  How approaches to teaching are affected by discipline and teaching context , 2006 .

[22]  R. Slavin.  Evidence-Based Education Policies: Transforming Educational Practice and Research , 2002 .

[23]  T. Cook.  Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community has Offered for not Doing Them , 2002 .

[24]  C. Rust.  The Impact of Assessment on Student Learning , 2002 .

[25]  Darrell P. Rowbottom,et al.  The Myth of ‘Scientific Method’ in Contemporary Educational Research , 2006 .

[26]  Mary Thorpe,et al.  Effective online interaction: Mapping course design to bridge from research to practice , 2008 .

[27]  M. Sundberg,et al.  Assessing student learning. , 2002, Cell biology education.

[28]  Adrian Kirkwood,et al.  Missing: evidence of a scholarly approach to teaching and learning with technology in higher education , 2013 .

[29]  D. Watkins,et al.  Awareness of the backwash effect of assessment: A phenomenographic study of the views of Hong Kong and Swedish lecturers , 2005 .