Towards Analytics for Wholistic School Improvement: Hierarchical Process Modelling and Evidence Visualization

Central to the mission of most educational institutions is the task of preparing the next generation of citizens to contribute to society. Schools, colleges, and universities value a range of outcomes — e.g., problem solving, creativity, collaboration, citizenship, service to community — as well as academic outcomes in traditional subjects. Often referred to as “wider outcomes,” these are hard to quantify. While new kinds of monitoring technologies and public datasets expand the possibilities for quantifying these indices, we need ways to bring that data together to support sense-making and decision-making. Taking a systems perspective, the hierarchical process modelling (HPM) approach and the “Perimeta” visual analytic provides a dashboard that informs leadership decision-making with heterogeneous, often incomplete evidence. We report a prototype of Perimeta modelling from education, aggregating wider outcomes data across a network of schools, and calculating their cumulative contribution to key performance indicators, using the visual analytic of the Italian flag to make explicit not only the supporting evidence, but also the challenging evidence, as well as areas of uncertainty. We discuss the nature of the modelling decisions and implicit values involved in quantifying these kinds of educational outcomes.

[1]  Halia Silins,et al.  Schools as Learning Organisations - Effects on Teacher Leadership and Student Outcomes , 2004 .

[2]  Yuha Jung.  A New Culture of Learning: Cultivating the Imagination for a World of Constant Change , 2015 .

[3]  M. Fullan,et al.  How the world ’ s best-performing school systems come out on top September , 2007 .

[4]  David Pedder,et al.  Evaluating the wider outcomes of schools: Complex systems modelling for leadership decisioning , 2017 .

[5]  Douglas M. Hawkins,et al.  Elements of Continuous Multivariate Analysis , 1969 .

[6]  A. Bryk.  Ruminations on Reinventing an R & D Capacity for Educational Improvement , 2008 .

[7]  Fowler Vincent Harper,et al.  Carnegie Foundation for the Advancement of Teaching , 1915, Nature.

[8]  M. Mason.  Complexity Theory and the Philosophy of Education , 2008 .

[9]  J. Bransford,et al.  Preparing Teachers for a Changing World: What Teachers Should Learn and Be Able to Do. , 2005 .

[10]  John MacBeath,et al.  Leadership for learning : international perspectives , 2008 .

[11]  Karen Littleton,et al.  Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space , 2014, J. Learn. Anal..

[12]  J. P. Davis,et al.  Managing Assets under Uncertainty , 2000 .

[13]  D. Snowden,et al.  A leader's framework for decision making , 2007 .

[14]  Jim W. Hall,et al.  A decision-support methodology for performance-based asset management , 2004 .

[15]  Daniel T. Hickey,et al.  Educational data sciences: framing emergent practices for analytics of learning, organizations, and systems , 2014, LAK.

[16]  John P. Campbell,et al.  Academic Analytics: A New Tool for a New Era. , 2007 .

[17]  Simon Buckingham Shum,et al.  Constructing Knowledge Art: An Experiential Perspective on Crafting Participatory Representations , 2014, Constructing Knowledge Art.

[18]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[19]  W. Cui,et al.  Interval probability theory for evidential support , 1990, Int. J. Intell. Syst..

[20]  C. Dweck.  Self-theories and goals: their role in motivation, personality, and development. , 1990, Nebraska Symposium on Motivation. Nebraska Symposium on Motivation.

[21]  John MacBeath,et al.  Self-Evaluation: What's In It For Schools? , 2002 .

[22]  C. Goldspink,et al.  A conceptual model and set of instruments for measuring student engagement in learning , 2013 .

[23]  Ruth Deakin Crick.  Learning Analytics: Layers, Loops and Processes in a Virtual Learning Infrastructure , 2017 .

[24]  Brent Davis,et al.  Complexity and Education: Inquiries Into Learning, Teaching, and Research , 2006 .

[25]  Wynne Harlen,et al.  Testing, motivation and learning , 2002 .

[26]  David Pedder.  Profiling teachers' professional learning practices and values: differences between and within schools , 2007 .

[27]  John MacBeath,et al.  How teachers value and practise professional learning , 2005 .

[28]  M. Bottery.  The challenges of educational leadership , 2004 .

[29]  Leah P. Macfadyen,et al.  Embracing Big Data in Complex Educational Systems: The Learning Analytics Imperative and the Policy Challenge. , 2014 .

[30]  Peter Checkland,et al.  Soft Systems Methodology in Action , 1990 .

[31]  J. R. Beaumont.  Diagnosing the System for Organizations , 1985 .

[32]  Danah Zohar.  Rewiring the Corporate Brain: Using the New Science to Rethink How We Structure and Lead Organizations , 1997 .

[33]  Mary James,et al.  Improving Learning How to Learn: Classrooms, Schools and Networks , 2007 .

[34]  D. Blockley,et al.  The importance of being process , 2010, Building Bridges.

[35]  Barbara L. McCombs,et al.  The School Leader's Guide to Learner-Centered Education: From Complexity to Simplicity , 2008 .

[36]  J. Brown,et al.  Learning for a World of Constant Change , 2009 .

[37]  Guy Claxton,et al.  Expanding the Capacity to Learn: A new end for education? , 2006 .

[38]  W. Harlen,et al.  Testing and Motivation for Learning , 2003 .

[39]  M. James,et al.  Broadening the basis of assessment to prevent the narrowing of learning , 1998 .

[40]  Simon Buckingham Shum,et al.  Learning Analytics for 21st Century Competencies , 2016, J. Learn. Anal..

[41]  Kenneth J. Rowe,et al.  The Impact of Leadership on Student Outcomes: An Analysis of the Differential Effects of Leadership Types , 2008 .

[42]  Stafford Beer,et al.  The Viable System Model : its provenance , development , methodology and pathology * Stafford Beer = President of the World Organization for Systems and Cybernetics , 2000 .

[43]  James W. Hall,et al.  Uncertain inference using interval probability theory , 1998, Int. J. Approx. Reason..

[44]  James M. McPartland,et al.  Organizing Schools for Improvement: Lessons from Chicago , 2011 .

[45]  Chris Goldspink,et al.  Developing Resilient Agency in Learning: The Internal Structure of Learning Power , 2015 .

[46]  D. Snowden,et al.  A leader's framework for decision making. A leader's framework for decision making. , 2007, Harvard business review.

[47]  David Pedder,et al.  The lost promise of teacher professional development in England , 2011 .

[48]  H. Gunter.  Leaders and leadership in education , 2001 .

[49]  David Blockley,et al.  Doing it Differently: Systems for Rethinking Construction , 2000 .

[50]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[51]  Ruth Deakin Crick,et al.  Citizenship, discourse ethics and an emancipatory model of lifelong learning , 2010 .

[52]  Peter Mortimore,et al.  Learning: The Treasure Within , 1999 .

[53]  C.J.H. Mann,et al.  Systems approaches to management , 2002 .

[54]  J. R. Beaumont.  Diagnosing the System for Organisations , 1986 .

[55]  Louis M. Gomez,et al.  Learning to Improve: How America’s Schools Can Get Better at Getting Better , 2015 .

[56]  Peter Checkland,et al.  Systems Thinking, Systems Practice , 1981 .

[57]  David Pedder.  Organizational conditions that foster successful classroom promotion of Learning How to Learn , 2006 .

[58]  R. Crick,et al.  Distributing Leadership for Personalizing Learning , 2007 .

[59]  Rebecca Ferguson,et al.  Setting learning analytics in context: overcoming the barriers to large-scale adoption , 2014, LAK '14.

[60]  L. White,et al.  Evaluating problem-structuring methods: developing an approach to show the value and effectiveness of PSMs , 2006, J. Oper. Res. Soc..

[61]  Kristen E. DiCerbo,et al.  Harnessing the Currents of the Digital Ocean , 2014 .

[62]  Louis M. Gomez,et al.  Getting Ideas into Action: Building Networked Improvement Communities in Education , 2010 .

[63]  D. Wiliam,et al.  ’I’ ll be a nothing’: structure, agency and the construction of identity through assessment , 1999 .

[64]  Ben Daniel,et al.  Big Data and analytics in higher education: Opportunities and challenges , 2015, Br. J. Educ. Technol..

[65]  Simon Buckingham Shum,et al.  Learning dispositions and transferable competencies: pedagogy, modelling and learning analytics , 2012, International Conference on Learning Analytics and Knowledge.

[66]  A. MacDonald,et al.  Problem-structuring methods and project management: an example of stakeholder involvement using Hierarchical Process Modelling methodology , 2010, J. Oper. Res. Soc..

[67]  B. McCombs,et al.  Children's and Teachers' Perceptions of Learner-Centered Practices, and Student Motivation: Implications for Early Schooling , 2008, The Elementary School Journal.

[68]  Lisa S. Blackwell,et al.  Implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and an intervention. , 2007, Child development.