A utilisation focussed and viable systems approach for evaluating technology supported learning

The paper uses a higher education case study to illustrate a participative theory of change approach to evaluating technology supported learning. The approach is informed by the Viable Systems Model (VSM) and utilisation-focussed evaluation and, falls within the tradition of facilitated modelling approaches to operational research. We argue that this approach worked well in engaging primary evaluation users in a process of collaborative action research to improving an educational development initiative and that the approach helped generate information relevant to answering its primary users’ questions, to inform their specific decisions and actions relevant to their quality enhancement responsibilities.

[1]  P. Mason,et al.  Constructing Theories of Change , 2007 .

[2]  Steven K. Van Dyke Dimensions of Quality , 1994 .

[3]  Susan Miller,et al.  Using Systems Thinking to Educate for Sustainability in a Business School , 2014, Syst..

[4]  Jonathan Stacks,et al.  Developmental Evaluation , 2011, Health promotion practice.

[5]  Mark Taylor,et al.  Supporting crime detection and operational planning with soft systems methodology and viable systems model , 2009 .

[6]  G. Midgley Systemic Intervention: Philosophy, Methodology, and Practice , 2000 .

[7]  Erik Duval,et al.  Learning Analytics Dashboard Applications , 2013 .

[8]  D. Houston,et al.  Knowledge, power and meanings shaping quality assurance in higher education: a systemic critique , 2013 .

[9]  Alan Woodley,et al.  National student feedback surveys in distance education: an investigation at the UK Open University , 2011 .

[10]  George Siemens,et al.  Learning Analytics , 2013 .

[11]  Ray Pawson,et al.  Evidence-based policy: a realist perspective. , 2006 .

[12]  Laurie P. Dringus,et al.  Learning Analytics Considered Harmful. , 2012 .

[13]  John B. Biggs,et al.  Teaching for Quality Learning at University: What the Student Does , 1999 .

[14]  Murray Saunders,et al.  Using Evaluation to Create ‘Provisional Stabilities’ , 2005 .

[15]  Chris L. S. Coryn,et al.  A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009 , 2011 .

[16]  Irene C. L. Ng,et al.  Viable Service Systems and Decision Making in Service Management , 2012 .

[17]  N. Selwyn,et al.  What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning , 2017 .

[18]  Gabi Diercks-O'Brien,et al.  Exploring Stakeholder Engagement in Impact Evaluation Planning in Educational Development Work , 2009 .

[19]  Demichelis,et al.  Evaluation of the , 1992, Physical review. B, Condensed matter.

[20]  D. Marks-Maran Educational research methods for researching innovations in teaching, learning and assessment: The nursing lecturer as researcher. , 2015, Nurse education in practice.

[21]  Nicolette Lee Systems models in educational research: a review and realignment in the context of curriculum , 2015 .

[22]  John Brocklesby,et al.  Using the Viable Systems Model to examine multi-agency arrangements for combatting transnational organised crime , 2012, J. Oper. Res. Soc..

[23]  Fran Ackermann,et al.  Getting "Messy" with Problems: The Challenges of Teaching "Soft" OR , 2011, INFORMS Trans. Educ..

[24]  Ricardo Abad Barros-Castro,et al.  A Systemic Framework for Evaluating Computer-Supported Collaborative Learning—Mathematical Problem-solving (CSCL-MPS) Initiatives: Insights from a Colombian Case , 2013, Systemic Practice and Action Research.

[25]  Lee Harvey,et al.  Student Feedback [1] , 2003 .

[26]  Huey-tsyh Chen Theory-driven evaluations , 1990 .

[27]  Carol H. Weiss Evaluation Research: Methods of Assessing Program Effectiveness , 1972 .

[28]  Anthony G. Picciano The Evolution of Big Data and Learning Analytics in American Higher Education , 2012 .

[29]  V. Bamber,et al.  Evaluating learning and teaching: institutional needs and individual practices , 2012 .

[30]  Martin Reynolds,et al.  Systems thinking and Equity-focused evaluations , 2012 .

[31]  M. Oliver,et al.  Technological determinism in educational technology research: some alternative ways of thinking about the relationship between learning and technology , 2011, J. Comput. Assist. Learn..

[32]  Lee Harvey,et al.  Evaluation for What? , 2002 .

[33]  Lee Harvey,et al.  Fifteen Years of Quality in Higher Education (Part Two) , 2010 .

[34]  M. Radford Researching classrooms: complexity and chaos , 2006 .

[35]  Shirley Alexander,et al.  Using Students' Experiences to Derive Quality in an e-Learning System: An Institution's Perspective , 2007, J. Educ. Technol. Soc..

[36]  E. Guba,et al.  Fourth Generation Evaluation , 1989 .

[37]  C. Weiss How Can Theory-Based Evaluation Make Greater Headway? , 1997 .

[38]  Dennis Sumara,et al.  Complexity science and educational action research: toward a pragmatics of transformation , 2005 .

[39]  Michael Quinn Patton,et al.  Essentials of utilization-focused evaluation , 2012 .

[40]  G. Westhorp Developing complexity-consistent theory in a realist investigation , 2013 .

[41]  Sung-Wan Kim,et al.  Validation of an evaluation model for learning management systems , 2008, J. Comput. Assist. Learn..

[42]  D. Kolb Experiential Learning: Experience as the Source of Learning and Development , 1983 .

[43]  Andrea Martínez,et al.  A methodology for supporting strategy implementation based on the VSM: A case study in a Latin-American multi-national , 2015, Eur. J. Oper. Res..

[44]  Linda Price,et al.  Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review , 2014 .

[45]  Ian Dunwell,et al.  Foundations of dynamic learning analytics: Using university student data to increase retention , 2015, Br. J. Educ. Technol..

[46]  Ann Jones,et al.  Open systems evaluation and the logic model: Program planning and evaluation tools , 1995 .

[47]  Fran Ackermann,et al.  Problem structuring methods 'in the Dock': Arguing the case for Soft OR , 2012, Eur. J. Oper. Res..

[48]  M. Oliver,et al.  Talking back to theory : the missed opportunities in learning technology research , 2011 .

[49]  Jon Walker,et al.  Complexity management in practice: A Viable System Model intervention in an Irish eco-community , 2013, Eur. J. Oper. Res..

[50]  Russ Vince,et al.  Organizing Reflection , 2002 .

[51]  Donald A. Schön,et al.  Organizational Learning II: Theory, Method, and Practice , 1995 .

[52]  Werner Ulrich,et al.  Critical heuristics of social systems design , 1987 .

[53]  Nikolaos K. Dimitriou,et al.  A cybernetic framework for viable virtual enterprises: The use of VSM and PSM systemic methodologies , 2006, Kybernetes.

[54]  Bülent,et al.  OPERATIONAL RESEARCH IN NARLIDERE EDUCATION RESEARCH AND HEALTH DISTRICT’S HEALTH CENTRES (1999-2002) - , 2007 .

[55]  James Williams,et al.  Fitness for Purpose? National and Institutional Approaches to Publicising the Student Voice , 2007 .

[56]  Shane Dawson,et al.  Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan , 2012, J. Educ. Technol. Soc..

[57]  Tom Prebble,et al.  Exploring Quality in a University Department: Perspectives and Meanings , 2008 .

[58]  Mhairi Mackenzie,et al.  Theories of Change and Realistic Evaluation , 2007 .

[59]  Haruo Hayashi,et al.  Using the Viable System Model (VSM) to structure information processing complexity in disaster response , 2013, Eur. J. Oper. Res..

[60]  John Fritz,et al.  Classroom walls that talk: Using online course activity data of successful students to raise self-awareness of underperforming peers , 2011, Internet High. Educ..

[61]  Utilising collaborative forms of educational action research: some reflections , 2007 .

[62]  Jon Walker,et al.  A complexity approach to sustainability - Stafford Beer revisited , 2008, Eur. J. Oper. Res..

[63]  S. Beer The Brain of the Firm , 1972 .

[64]  Haruo Hayashi,et al.  Application of the Viable System Model to analyse communications structures: A case study of disaster response in Japan , 2015, Eur. J. Oper. Res..

[65]  D. Houston Systemic Intervention in a University Department: Reflections on Arrested Action Research , 2008 .

[66]  Margaret J. Cox,et al.  Effects of ICT: Do we know what we should know? , 2007, Education and Information Technologies.

[67]  Alberto Paucar-Caceres,et al.  A Soft Systems Methodology (SSM) Based Framework for Evaluating Managed Learning Environments , 2011 .

[68]  Ana M Mogus,et al.  The impact of student activity in a virtual learning environment on their final mark , 2012 .

[69]  Peter Checkland,et al.  Diagnosing the system for organizations: S. BEER Wiley, Chichester, 1985, 152 + xiii pages, £7.50 , 1986 .

[70]  Paul R. Trowler,et al.  Freeing the chi of change: the Higher Education Academy and enhancing teaching and learning in higher education , 2005 .

[71]  Raul Espejo,et al.  The Viable System Model as a Framework for Understanding Organizations , 1997 .

[72]  Gaby Jacobs ‘A guided walk in the woods’: boundary crossing in a collaborative action research project , 2017 .

[73]  D. Schoen,et al.  The Reflective Practitioner: How Professionals Think in Action , 1985 .

[74]  James Owen Drife,et al.  Student feedback , 2000, BMJ : British Medical Journal.

[75]  Michael Jackson,et al.  A systemic look at educational development programs: two perspectives on a recent Colombian experience , 2002 .

[76]  John Mayne,et al.  Contribution analysis: Coming of age? , 2012 .

[77]  Catherine D. Bruce,et al.  Examining what we mean by collaboration in collaborative action research: a cross-case analysis , 2011 .

[78]  Ali Azadeh,et al.  Diagnosing, Simulating and Improving Business Process Using Cybernetic Laws and the Viable System Model: The Case of a Purchasing Process , 2012 .

[79]  C. S. Nair,et al.  Student Engagement: The Key to Improving Survey Response Rates , 2008 .

[80]  John Stephens,et al.  A Set of Conventions, a Model: An Application of Stafford Beer’s Viable Systems Model to the Strategic Planning Process , 2011 .

[81]  Sean Devine,et al.  The Viable Systems Model Applied to a National System of Innovation to Inform Policy Development , 2005 .

[82]  L. Alberto Franco,et al.  Facilitated modelling in operational research , 2010, Eur. J. Oper. Res..

[83]  Deborah Compeau,et al.  The Effects of Self-Regulated Learning Processes on E-Learning Outcomes in Organizational Settings , 2012, J. Manag. Inf. Syst..

[84]  L. Harvey A History and Critique of Quality Evaluation in the UK. , 2005 .

[85]  Shauna L. Mahajan,et al.  Systems thinking for planning and evaluating conservation interventions , 2019, Conservation Science and Practice.

[86]  Steve May,et al.  Student retention in higher education: what role for virtual learning environments? , 2009 .

[87]  David Kember Long-term outcomes of educational action research projects , 2002 .

[88]  John Biggs,et al.  The reflective institution: Assuring and enhancing the quality of teaching and learning , 2001 .

[89]  D. Houston TQM and Higher Education: A Critical Systems Perspective on Fitness for Purpose , 2007 .

[90]  Russell Clemens,et al.  Environmental Scanning and Scenario Planning: A 12 month Perspective on Applying the Viable Systems Model to Developing Public Sector Foresight , 2009 .

[91]  F. Leeuw,et al.  Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation , 2010 .

[92]  Lee Harvey,et al.  Fifteen Years of Quality in Higher Education , 2010 .

[93]  E. Jantsch The design of inquiring systems,: by C. West Churchman New York, Basic Books, 1972. , 1972 .

[94]  D. Houston,et al.  Rethinking quality and improvement in higher education , 2008 .

[95]  G. Westhorp,et al.  Using complexity-consistent theory for evaluating complex systems , 2012 .