Building the foundations for measuring learning gain in higher education: a conceptual framework and measurement instrument

ABSTRACT In this paper, we set out the first step towards the measurement of learning gain in higher education by putting forward a conceptual framework for understanding learning gain that is relevant across disciplines. We then introduce the operationalisation of this conceptual framework into a new set of measurement tools. With the use of data from a large-scale survey of 11 English universities and over 4,500 students, we test the reliability and validity of the measurement instrument empirically. We find support in the data for the reliability of most of the measurement scales we put forward, as well as for the validity of the conceptual framework. Based on these results, we reflect on the conceptual framework and associated measurement tools in the context of at-scale deployment and the potential implications for policy and practice in higher education.

[1]  Daniel L. Dinsmore.  Toward a Dynamic, Multidimensional Research Framework for Strategic Processing , 2017 .

[2]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[3]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[4]  Karen L. Gischlar,et al.  Item Construction Using Reflective, Formative, or Rasch Measurement Models: Implications for Group Work , 2017 .

[5]  Tom Booth,et al.  The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development , 2018 .

[6]  P. Pintrich.  A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students , 2004 .

[7]  Jichuan Wang,et al.  Applying structural equation modelling in educational research / La aplicación del modelo de ecuación estructural en las investigaciones educativas , 2017 .

[8]  Alison Cotgrave,et al.  Measuring student learning gain: a review of transatlantic measurements of assessments in higher education , 2017 .

[9]  Markku Niemivirta,et al.  Relations between teacher students’ approaches to learning, cognitive and attributional strategies, well-being, and study success , 2012 .

[10]  Jesús M. Alvarado,et al.  Best Alternatives to Cronbach's Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements , 2016, Front. Psychol..

[11]  Amy L. Dent,et al.  The Relation Between Self-Regulated Learning and Academic Achievement Across Childhood and Adolescence: A Meta-Analysis , 2016 .

[12]  Akane Zusho,et al.  Toward an Integrated Model of Student Learning in the College Classroom , 2017 .

[13]  Luke K. Fryer,et al.  Reciprocal modelling of Japanese university students' regulation strategies and motivational deficits for studying , 2016 .

[14]  J. Vermunt,et al.  Consistency and variability of learning strategies in different university courses , 1999 .

[15]  F. Pina,et al.  Creencias epostemológicas y de aprendizaje en la formación inicial de profesores , 2012 .

[16]  H. Goldstein,et al.  The limitations of using school league tables to inform school choice , 2009 .

[17]  Icek Ajzen,et al.  From Intentions to Actions: A Theory of Planned Behavior , 1985 .

[18]  F. Dochy,et al.  Using student-centred learning environments to stimulate deep approaches to learning: Factors encouraging or discouraging their effectiveness , 2010 .

[19]  Nicholas A. Bowman.  Can 1st-Year College Students Accurately Report Their Learning and Development? , 2010 .

[20]  Timothy Rodgers.  Measuring Value Added in Higher Education: A Proposed Methodology for Developing a Performance Indicator Based on the Economic Value Added to Graduates , 2007 .

[21]  Marc A. Brackett,et al.  Predicting school success: Comparing Conscientiousness, Grit, and Emotion Regulation Ability , 2014 .

[22]  D. Dill,et al.  Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems , 2005 .

[23]  N. Shephard,et al.  How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic background , 2016 .

[24]  Harvey Goldstein,et al.  The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’ , 2017 .

[25]  Roderick P. McDonald,et al.  The dimensionality of tests and items , 1981 .

[26]  Higher education: core skills in a learning society , 1997 .

[27]  L. Unwin,et al.  Approaches to learning , 2002 .

[28]  J. Vermunt,et al.  Patterns in Student Learning: Relationships Between Learning Strategies, Conceptions of Learning, and Learning Orientations , 2004 .

[29]  George D. Kuh,et al.  Student Engagement and Student Learning: Testing the Linkages* , 2006 .

[30]  Eric M. Sobel,et al.  Using League Table Rankings in Public Policy Formation : Statistical Issues , 2014 .

[31]  M. W. Allen.  The goals of universities , 1988 .

[32]  N. Smith,et al.  Evaluating engagement with graduate outcomes across higher education institutions in Aotearoa/New Zealand , 2015 .

[33]  C. MacCann,et al.  Do time management, grit, and self-control relate to academic achievement independently of conscientiousness? , 2010 .

[34]  George D. Kuh.  What We're Learning About Student Engagement From NSSE: Benchmarks for Effective Educational Practices , 2003 .

[35]  R. Blundell,et al.  The Returns to Higher Education in Britain: Evidence From a British Cohort , 2000 .

[36]  Jichuan Wang,et al.  Structural Equation Modeling: Applications Using Mplus , 2012 .

[37]  Ernest T. Pascarella,et al.  Does independent research with a faculty member enhance four-year graduation and graduate/professional degree plans? Convergent results with different analytical methods , 2016 .

[38]  J. Biggs.  Student Approaches to Learning and Studying , 1987 .

[39]  Ernest T. Pascarella,et al.  How college affects students : findings and insights from twenty years of research , 1992 .

[40]  Velda McCune,et al.  The Conceptual Bases of Study Strategy Inventories , 2004 .

[41]  Luke K. Fryer,et al.  Supporting interest in a study domain: A longitudinal test of the interplay between interest, utility-value, and competence beliefs , 2017, Learning and Instruction.

[42]  Sample Size Planning for Confirmatory Factor Models: Power and Accuracy for Effects of Interest , 2018 .

[43]  J. Ponterotto,et al.  An Overview of Coefficient Alpha and a Reliability Matrix for Estimating Adequacy of Internal Consistency Coefficients with Psychological Research Measures , 2007, Perceptual and motor skills.

[44]  Timothy Rodgers.  Measuring Value Added in Higher Education: Do Any of the Recent Experiences in Secondary Education in the United Kingdom Suggest a Way Forward?. , 2005 .

[45]  E. Boyle,et al.  Learning styles and academic outcome: the validity and utility of Vermunt's Inventory of Learning Styles in a British higher education setting. , 2003, The British journal of educational psychology.

[46]  P. Petegem,et al.  Analysing Change in Learning Strategies over Time: A Comparison of Three Statistical Techniques. , 2013 .

[47]  Harry Brighouse,et al.  The Aims of Higher Education: Problems of Morality and Justice , 2015 .

[48]  S. Barrie,et al.  A conceptual framework for the teaching and learning of generic graduate attributes , 2007 .

[49]  Jan H. F. Meyer.  The modelling of ‘dissonant’ study orchestration in higher education , 2000 .

[50]  Roger Bolus,et al.  The Collegiate Learning Assessment , 2007, Evaluation review.

[51]  Paul D. Umbach,et al.  Student Survey Response Rates across Institutions: Why Do they Vary? , 2006 .

[52]  William Revelle,et al.  The international cognitive ability resource: Development and initial validation of a public-domain measure , 2014 .

[53]  Franziska Perels,et al.  Self-regulated learning profiles in college students: Their relationship to achievement, personality, and the effectiveness of an intervention to foster self-regulated learning , 2016 .

[54]  Günter Krampen,et al.  The differential development of epistemic beliefs in psychology and computer science students: A four-wave longitudinal study , 2017 .

[55]  Tapabrata Maiti,et al.  Principles and Practice of Structural Equation Modeling (2nd ed.) , 2006 .

[56]  Angela L. Duckworth,et al.  Grit: perseverance and passion for long-term goals. , 2007, Journal of personality and social psychology.

[57]  Beverley Oliver,et al.  Graduate attributes as a focus for institution-wide curriculum renewal: innovations and challenges , 2013 .

[58]  Gregory J. Boyle,et al.  Does item homogeneity indicate internal consistency or item redundancy in psychometric scales , 1991 .

[59]  Nicholas A. Bowman.  Disequilibrium and Resolution: The Nonlinear Effects of Diversity Courses on Well-Being and Orientations toward Diversity , 2010 .

[60]  Brian P. An.  The Role of Academic Motivation and Engagement on the Relationship between Dual Enrollment and Academic Performance , 2014 .

[61]  I. Mammarella,et al.  Mathematics Anxiety, Working Memory, and Mathematics Performance in Secondary-School Children , 2016, Front. Psychol..

[62]  Alexander W. Astin,et al.  Involvement in Learning Revisited: Lessons We Have Learned. , 1999 .

[63]  David Gijbels,et al.  Do Students Develop Towards More Deep Approaches to Learning During Studies? A Systematic Review on the Development of Students’ Deep and Surface Approaches to Learning in Higher Education , 2017, Educational Psychology Review.

[64]  Alexander W. Astin,et al.  How Risky Are One-Shot Cross-Sectional Assessments of Undergraduate Students? , 2003 .

[65]  Robin Naylor,et al.  Determinants of Degree Performance in UK Universities: A Statistical Analysis of the 1993 Student Cohort , 2001 .

[66]  R. Cattell.  The Scientific Use of Factor Analysis in Behavioral and Life Sciences , 2012 .

[67]  W. G. Perry.  Forms of Intellectual and Ethical Development in the College Years: A Scheme. Jossey-Bass Higher and Adult Education Series. , 1970 .

[68]  Jennifer A. Fredricks,et al.  Using qualitative methods to develop a survey measure of math and science engagement , 2016 .

[69]  M. Miller,et al.  Sample Size Requirements for Structural Equation Models , 2013, Educational and psychological measurement.

[70]  Matthew J. Mayhew,et al.  Moral Judgement Development in Higher Education: Insights from the Defining Issues Test , 2002 .

[71]  Benoit Guerin,et al.  Learning Gain in Higher Education , 2021, International Perspectives on Higher Education Research.

[72]  F. Marton,et al.  Approaches to learning , 2003 .

[73]  Serge Herzog.  Gauging Academic Growth of Bachelor Degree Recipients: Longitudinal vs. Self-Reported Gains in General Education. , 2011 .

[74]  Karen W Bauer,et al.  The Effect of Personality and Precollege Characteristics on First-Year Activities and Academic Performance , 2003 .

[75]  Michael K. Filsecker,et al.  Student engagement, context, and adjustment: Addressing definitional, measurement, and methodological issues , 2016 .

[76]  M. Schommer-Aikins,et al.  Epistemological and learning beliefs of trainee teachers studying education , 2012 .

[77]  Huili Liu,et al.  Investigating student learning gains in college: a longitudinal study† , 2017 .

[78]  Bill Kules,et al.  Computational thinking is critical thinking: Connecting to university discourse, goals, and learning outcomes , 2016, ASIST.

[79]  Marcia J. Simmering,et al.  A Tale of Three Perspectives , 2009 .

[80]  C. Vleuten,et al.  Fifteen Years of Experience with Progress Testing in a Problem-Based Learning Curriculum. , 1996 .

[81]  J. Beishuizen,et al.  The relation between intellectual and metacognitive skills from a developmental perspective , 2004 .

[82]  Feifei Ye,et al.  The Math and Science Engagement Scales: Scale development, validation, and psychometric properties , 2016 .

[83]  Stephen R. Porter,et al.  The Impact of Lottery Incentives on Student Survey Response Rates , 2003 .

[84]  S. Barrie.  A research‐based approach to generic graduate attributes policy , 2004 .

[85]  Bengt Muthén,et al.  Number of Subjects and Time Points Needed for Multilevel Time-Series Analysis: A Simulation Study of Dynamic Structural Equation Modeling , 2018 .

[86]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[87]  R. Cattell,et al.  Measurement and Statistical Models in the Study of Personality and Intelligence , 1995 .

[88]  J. F. Soares,et al.  Do Provão ao ENADE: uma análise comparativa dos exames nacionais utilizados no Ensino Superior Brasileiro , 2006 .

[89]  Maryam Hussain,et al.  Investigating grit and its relations with college students’ self-regulated learning and academic achievement , 2015 .

[90]  Matthew J. Mayhew.  A Multilevel Examination of the Influence of Institutional Type on the Moral Reasoning Development of First-Year Students , 2012 .

[91]  Angela L. Duckworth,et al.  Development and Validation of the Short Grit Scale (Grit–S) , 2009, Journal of personality assessment.

[92]  Duncan David Nulty,et al.  The adequacy of response rates to online and paper surveys: what can be done? , 2008 .

[93]  A. Satorra,et al.  Power of the likelihood ratio test in covariance structure analysis , 1985 .

[94]  M. Schommer-Aikins,et al.  Ways of Knowing and Willingness to Argue , 2009, The Journal of psychology.

[95]  Hamish Coates,et al.  An international assessment of bachelor degree graduates' learning outcomes , 2012 .

[96]  O. Liu.  Measuring value‐added in higher education: conditions and caveats – results from using the Measure of Academic Proficiency and Progress (MAPP™) , 2011 .

[97]  Mantz Yorke,et al.  Employability and Good Learning in Higher Education , 2003 .

[98]  Outcomes, Testing, Learning: What’s at Stake? , 2014 .

[99]  Ernest T. Pascarella,et al.  How Effective are the NSSE Benchmarks in Predicting Important Educational Outcomes? , 2010 .

[100]  Ernest T. Pascarella,et al.  How College Affects Students: A Third Decade of Research. Volume 2. , 2005 .

[101]  P. Tynjälä.  Writing, Learning And The Development Of Expertise In Higher Education , 2001 .

[102]  P. Murtaugh,et al.  PREDICTING THE RETENTION OF UNIVERSITY STUDENTS , 1999 .

[103]  Mark H. Salisbury,et al.  Student perceptions of effective instruction and the development of critical thinking: a replication and extension , 2015 .

[104]  Donald D. Carpenter,et al.  The Theory of Planned Behavior as a Model of Academic Dishonesty in Engineering and Humanities Undergraduates , 2007 .

[105]  Vincent Donche,et al.  A Learning Patterns Perspective on Student Learning in Higher Education: State of the Art and Moving Forward , 2017, Educational Psychology Review.

[106]  P. Ewell.  The US National Survey of Student Engagement (NSSE) , 2010 .

[107]  Neville Bennett,et al.  Patterns of core and generic skill provision in higher education , 1999 .

[108]  Jeremy P. Smith,et al.  Graduate Employability: Policy and Performance in Higher Education in the UK , 2000 .

[109]  Susan M. Brookhart,et al.  Understanding Middle Students' Beliefs About Knowledge and Learning Using a Multidimensional Paradigm , 2000 .

[110]  Beverley Oliver,et al.  Want students to engage? Contextualise graduate learning outcomes and assess for employability , 2018 .

[111]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[112]  Trends in examination performance and exposure to standardised tests in England and Wales , 2016 .

[113]  K. Lonka,et al.  Aspects and Prospects of Measuring Studying and Learning in Higher Education , 2004 .

[114]  Wayne J. Camara,et al.  Defining and Measuring College and Career Readiness: A Validation Framework , 2013 .

[115]  Helen Walkington,et al.  Graduate attributes: implications for higher education practice and policy , 2016 .

[116]  J. Vaske,et al.  Rethinking Internal Consistency in Cronbach's Alpha , 2017 .

[117]  K. Hakkarainen,et al.  How to measure PhD. students' conceptions of academic writing - and are they related to well-being? , 2014 .